Methods, devices, and systems for displaying a user interface on a user and detecting touch gestures

ABSTRACT

An example method of identifying a touch gesture on a user is provided. The method includes receiving, by one or more transducers of a wearable device attached to an appendage of the user, a set of signals that propagate through the user&#39;s appendage and establish a signal pathway to the wearable device. The method also includes, while receiving the set of signals, determining baseline characteristics for the signal pathway, and sensing a change in the baseline characteristics caused by user interaction with an affordance of a user interface projected or perceived on the user&#39;s appendage. The method further includes, in accordance with a determination that the sensed change in the baseline characteristics satisfies a contact criterion, reporting a candidate touch event on the user&#39;s appendage to a separate electronic device that creates the user interface or is in communication with another electronic device that creates the user interface.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/241,893, filed Jan. 7, 2019, entitled “Methods, Devices, and Systemsfor Displaying a User Interface on a User and Detecting Touch Gestures,”which claims priority to U.S. Provisional Application No. 62/647,559,filed Mar. 23, 2018, entitled “Methods, Devices, and Systems forDetermining Contact On a User of a Virtual Reality and/or AugmentedReality Device” and U.S. Provisional Application No. 62/647,560, filedMar. 23, 2018, entitled “Methods, Devices, and Systems for Projecting anImage Onto a User and Detecting Touch Gestures”, each of which isincorporated by reference herein in its entirety.

This application is related to U.S. Utility patent application Ser. No.16/241,871, filed Jan. 7, 2019, entitled “Methods, Devices, and Systemsfor Creating Haptic Stimulations and Tracking Motion of a User,” U.S.Utility patent application Ser. No. 16/241,890, filed Jan. 7, 2019,entitled “Methods, Devices, and Systems for Determining Contact On aUser of a Virtual Reality and/or Augmented Reality Device,” and U.S.Utility patent application Ser. No. 16/241,900, filed Jan. 7, 2019,entitled “Methods, Devices, and Systems for Creating Localized HapticStimulations on a User,” each of which is incorporated by referenceherein in its entirety.

TECHNICAL FIELD

This relates generally to virtual reality/augmented reality, includingbut not limited to projecting images onto a user and detecting gestureson the user relating to the projection.

BACKGROUND

Virtual reality (VR) and/or augmented reality (AR) technologies allowusers to interact with technologies in different ways. VR and/or ARallows a user to tactilely interact with the digital world. Users mayexperience haptic responses from electronic devices, allowing users arich experience. Wearable devices for VR and/or AR may allow users tointeract with the digital world through a medium distinct from anelectronic device's screen (e.g., a wearable device projects an imageonto a user's forearm using, e.g., augmented reality). However,determining a location of a gesture on the projected image withsufficient precision presents a challenge.

SUMMARY

Accordingly, there is a need for methods, devices, and systems forprojecting virtual images onto a user with sufficient fidelity indetermining whether a contact or gesture has occurred. One solution isto combine computer vision (e.g., a camera on a wearable device) and aseparate modality (e.g., a wearable wristband having one or moretransducers) for increased fidelity in determining a location and/orpressure of a gesture (e.g., contact).

In some embodiments, the solution explained above can be implemented ona wearable device that includes a plurality of transducers (e.g.,actuators). The wearable device in some instances is worn on the user'swrist (or various other body parts) and is used to project an image ontoa portion of the user's body, essentially creating a virtual oraugmented reality display on the user's body. In some embodiments, thewearable device may virtualize an image to be seen through a lens of thewearable device as though the image were projected onto the user.Moreover, the wearable device can be in communication with a host system(e.g., a virtual reality device and/or an augmented reality device,among others), and the wearable device can display images based oninstructions from the host system. As an example, the host system maydisplay video data to a user (e.g., may instruct a head-mounted displayto display the video data), and the host system may also instruct thewearable device to project images from the video onto the user's body.

The devices, systems, and methods described herein provide benefitsincluding but not limited to: (i) detecting a touch gesture on aprojected and/or virtual image by an appendage of a user, (ii)determining a location of a touch gesture on a projected image on auser's body, (iii) the wearable device does not encumber free motion ofa user's hand and/or wrist (or other body parts), and (iv) multiplewearable devices can be used simultaneously.

(A1) In accordance with some embodiments, a method is performed at afirst wearable device that includes a projector and a plurality oftransducers. The method includes projecting an image onto a portion of afirst appendage of a user of the first wearable device and detecting atouch gesture on the image by a second appendage of the user distinctfrom the first appendage. The method further includes at a secondwearable device having a camera and a processor, determining a locationof the touch gesture on the image where a computer system is instructedto perform an operation in accordance with the detecting and thelocation. In some embodiments, the first wearable device is attached toan appendage (e.g., wrist, forearm, bicep, thigh, ankle, etc.) of theuser and the second wearable device is worn on the head of the user(e.g., head-mounted display).

(A2) In some embodiments of the method of A1, further including, at thesecond wearable device, confirming, via the camera and the processor,that the detected touch gesture has occurred on the image by the secondappendage of the user. The computer system is instructed to perform theoperation in further accordance with the confirming.

(A3) In some embodiments of the method of any of A1-A2, the plurality oftransducers is a first plurality of transducers that can each generateone or more signals and the first wearable device further comprises afirst control circuit coupled to the first plurality of transducers.Moreover, the method further includes generating, via the firstplurality of transducers, signals that couple/vibrate into at least aportion of the first appendage of the user of the first wearable device.

(A4) In some embodiments of the method of A3, further includingreceiving, via a second plurality of transducers of a third wearabledevice, at least a portion of the signals generated by the firstplurality of transducers when the first appendage of the user is withina threshold distance from the third wearable device, wherein the user iswearing the third wearable device on a second appendage. The method alsoincludes in response to the receiving, determining, via a second controlcircuit of the third wearable device, a position of a portion of thefirst appendage with respect to a position of the third wearable device.The computer system is instructed to perform an operation in accordancewith the detecting, the position, and the location.

(A5) In some embodiments of the method of any of A1-A4, the touchgesture is a swipe gesture.

(A6) In some embodiments of the method of any of A1-A4, the touchgesture is a tap gesture.

(A7) In some embodiments of the method of any of A1-A4, the touchgesture is a pinch gesture.

(A8) In some embodiments of the method of any of A1-A7, the image is avideo stream.

(A9) In some embodiments of the method of any of A1-A8, the firstappendage is a first arm of the user and the second appendage is asecond arm of the user.

(A10) In another aspect, a system is provided that includes a firstwearable device, a second wearable device, a third wearable device, anda computer system, and the system is configured to perform the methodsteps described above in any of A1-A9.

(A11) In yet another aspect, one or more wearable devices are providedand the one or more wearable devices include means for performing themethod described in any one of A1-A9.

(A12) In still another aspect, a non-transitory computer-readablestorage medium is provided (e.g., as a memory device, such as externalor internal storage, that is in communication with a wearable device).The non-transitory computer-readable storage medium stores executableinstructions that, when executed by a wearable device with one or moreprocessors/cores, cause the wearable device to perform the methoddescribed in any one of A1-A9.

(B1) In accordance with some embodiments, another method is performed ata first wearable device, attached to a first appendage of a user, thatincludes one or more transducers. The method includes receiving, by theone or more transducers, a set of signals transmitted by a secondwearable device attached to the user, wherein (i) receiving the set ofsignals creates a signal pathway between the first and second wearabledevices, and (ii) signals in the set of signals propagate through atleast the user's first appendage. The method also includes determiningbaseline characteristics for the signal pathway created between thefirst wearable device and the second wearable device and sensing achange in the baseline characteristics while receiving the set ofsignals. The method also includes, in accordance with a determinationthat the sensed change in the baseline characteristics for the signalpathway satisfies a contact criterion, reporting a candidate touch eventon the user's first appendage. In some embodiments, the contactcriterion includes a touch criterion and a hover criterion. In suchembodiments, a sensed change in the baseline characteristics caused byfinger hovering event may satisfy the hover criterion but will notsatisfy the touch criterion.

(B2) In some embodiments of the method of B1, reporting the candidatetouch event comprises sending transducer data corresponding to thesensed change in the baseline characteristics to a computer system. Insome embodiments, the transducer data also includes a time stamp of whenthe sensed change in the baseline characteristics occurred. In someembodiments, reporting the candidate touch event includes sending, tothe computer system, a message reporting the candidate touch event.

(B3) In some embodiments of the method of B2, the computer systemdisplays, on the user's first appendage, a user interface that includesone or more affordances, and the candidate touch event reported by thefirst wearable device is associated with a first affordance of the oneor more affordances included in the user interface.

(B4) In some embodiments of the method of any of B2-B3, furtherincluding determining an approximate location of the candidate touchevent on the user's first appendage based, at least in part, on thesensed change in the baseline characteristics. The transducer data sentto the computer system further comprises information indicating theapproximate location of the candidate touch event.

(B5) In some embodiments of the method of B2, the transducer data sentto the computer system also indicates an approximate location of thecandidate touch event on the user's first appendage.

(B6) In some embodiments of the method of any of B3-B5, the computersystem: (i) captures, via one or more cameras, the candidate touchevent, (ii) generates image data according to the capturing of thecandidate touch event, and (iii) executes a function associated with thefirst affordance of the user interface after processing the transducerdata and the image data.

(B7) In some embodiments of the method of any of B1-B6, the baselinecharacteristics include a baseline phase value. Furthermore, sensing thechange in the baseline characteristics for the signal pathway comprisesdetecting a phase value of the signal pathway that differs from thebaseline phase value.

(B8) In some embodiments of the method of B7, the contact criterionincludes a phase difference threshold. Furthermore, reporting thecandidate touch event is performed in accordance with a determinationthat a difference between the phase value and the baseline phase valuesatisfies the phase difference threshold.

(B9) In some embodiments of the method of any of B1-B8, the baselinecharacteristics include a baseline amplitude value. Furthermore, sensingthe change in the baseline characteristics for the signal pathwaycomprises detecting an amplitude value of the signal pathway thatdiffers from the baseline amplitude value.

(B10) In some embodiments of the method of B9, the contact criterionincludes an amplitude difference threshold. Furthermore, reporting thecandidate touch event is performed in accordance with a determinationthat a difference between the amplitude value and the baseline amplitudevalue satisfies the amplitude difference threshold.

(B11) In some embodiments of the method of any of B1-B10, the baselinecharacteristics include a baseline amplitude value and a baseline phasevalue. Furthermore, sensing the change in the baseline characteristicsfor the signal pathway comprises detecting (i) an amplitude value of thesignal pathway that differs from the baseline amplitude value, and (ii)a phase value of the signal pathway that differs from the baseline phasevalue.

(B12) In some embodiments of the method of B11, the contact criterionincludes an amplitude difference threshold and a phase differencethreshold. Furthermore, reporting the candidate touch event is performedin accordance with a determination that: (i) a difference between theamplitude value and the baseline amplitude value satisfies the amplitudedifference threshold, and (ii) a difference between the phase value andthe baseline phase value satisfies the phase difference threshold.

(B13) In some embodiments of the method of any of B1-B12, the contactcriterion includes a time threshold. Furthermore, sensing the change inthe baseline characteristics comprises sensing the change for a periodof time and reporting the candidate touch event is performed inaccordance with a determination that the period of time satisfies thetime threshold. Alternatively, in some embodiments, the first wearabledevice continually sends transducer data to the computer device.

(B14) In some embodiments of the method of any of B1-B13, furtherincluding, before receiving the set of signals, receiving a pluralitypredetermined values for signals characteristics. Each of thepredetermined values for the signals characteristics corresponds to aspecific location of the first appendage of the user. In someembodiments, the transducer data of (B2) includes signalscharacteristics (e.g., values of phase, amplitude, etc.) thatsubstantially match one of the plurality predetermined values forsignals characteristics.

(B15) In some embodiments of the method of any of B1-B14, the candidatetouch event is selected from the group consisting of a tap gesture,press-and-holder gesture, a swipe gesture, a drag gesture, a multi-tapgesture, a pinch gesture, a pull gesture, and a twist gesture.

(B16) In some embodiments of the method of any of B1-B15, reporting thecandidate touch event comprises sending, to a computer system, dataassociated with the sensed change in the signal pathway, and thecomputer system determines whether the user intended to interact with anaffordance of a user interface displayed on the user's first appendagebased, at least in part, on the data associated with the sensed changein the signal pathway. For example, the computer system displays theuser interface on the user's first appendage, and the candidate touchevent reported by the first wearable device is associated with one ofthe affordances included in the user interface.

(B17) In some embodiments of the method of B16, the computer system (i)captures, via one or more cameras, an approximate location of thecandidate touch event, the approximate location of the candidate touchevent corresponding to a location of the affordance in the userinterface displayed on the user's first appendage, and (ii) executes afunction associated with the affordance in response to determining thatthe user intended to interact with the first affordance and inaccordance with the approximate location of the candidate touch event.

(B18) In some embodiments of the method of any of B1-B17, the computersystem is an artificial-reality system selected from the groupconsisting of an augmented-reality system, a virtual-reality system, anda mixed-reality system.

(B19) In yet another aspect, a wearable device is provided and thewearable device includes means for performing the method described inany one of B1-B18 and F1-F2.

(B20) In another aspect, a wearable device that includes one or moretransducers is provided. In some embodiments, the wearable device is incommunication with one or more processors and memory storing one or moreprograms which, when executed by the one or more processors, cause thewearable device to perform the method described in any one of B1-B18 andF1-F2.

(B21) In still another aspect, a non-transitory computer-readablestorage medium is provided (e.g., as a memory device, such as externalor internal storage, that is in communication with a wearable device).The non-transitory computer-readable storage medium stores executableinstructions that, when executed by a wearable device with one or moreprocessors/cores, cause the wearable device to perform the methoddescribed in any one of B1-B18 and F1-F2.

(B22) In still another aspect, a system is provided. The system includesa first wearable device, a second wearable device, and a computer systemthat are configured to perform the method described in any one of B1-B18and F1-F2. In some embodiments, the second wearable device and thecomputer system are part of the same device while in other embodimentsthey are separate devices.

(C1) In accordance with some embodiments, another method is performed atan artificial-reality system (e.g., AR system 1200, FIG. 12; VR system1300, FIG. 13), worn by a user, that includes a head-mounted display,one or more cameras, and at least one processor. The method includes (i)providing first instructions to the head-mounted display to display auser interface on a first appendage of the user, wherein the user isalso wearing, on a first appendage, a first wearable device that is incommunication with the artificial-reality system, and (ii) providingsecond instructions to a second wearable device to emit one or moresignals, wherein the one or more signals propagate through at least thefirst appendage of the user and are received by the first wearabledevice, thereby creating a signal pathway between the first wearabledevice and the second wearable device. The method also includes (i)receiving, from the first wearable device, data associated with thesignal pathway created between the first wearable device and the secondwearable device, and (ii) capturing, by the one or more cameras, acandidate touch event at a location on the user's first appendage,wherein the location is associated with an affordance of the userinterface. Thereafter, the method includes determining whether the userintended to interact with the affordance of the user interface based, atleast in part, on the data associated with the signal pathway, and inresponse to determining that the user intended to interact with theaffordance and in accordance with the captured location of the candidatetouch event, executing a function associated with the affordance.

(C2) In some embodiments of the method of C1, displaying the userinterface on the first appendage of the user includes: (i) projectingthe user interface on the user's first appendage, or (ii) presenting,using augmented reality, the user interface on the head-mounted displayso that the user perceives the user interface on the first appendage.

(D1) In accordance with some embodiments, another method is performed atan artificial-reality system (e.g., AR system 1200, FIG. 12; VR system1300, FIG. 13), worn by a user, that includes a head-mounted display,one or more cameras, and at least one processor. The method includes,while displaying a user interface on a first appendage of the user: (i)capturing, via the one or more cameras, a candidate touch event at alocation on a user's first appendage, wherein the location is associatedwith an affordance of the user interface, and (ii) receiving, from afirst wearable device worn by the user, data associated with thecandidate touch event, wherein the first wearable device is attached tothe user's first appendage. The method also includes determining whetherthe user's first appendage was touched based at least in part on thereceived data, and in accordance with a determination that the user'sfirst appendage was touched and based on the captured location of thecandidate touch event, executing a function associated with theaffordance of the user interface.

(D2) In some embodiments of the method of D1, the user interface isdisplayed on the first appendage of the user by: (i) projecting the userinterface on the user's first appendage, or (ii) presenting, usingaugmented reality, the user interface on the head-mounted display sothat the user perceives the user interface on the first appendage.

(D3) In some embodiments of the method of any of D1-D2, the firstwearable device performs the method described in any one of B1-B15 togenerate the data received by the artificial-reality system.

(E1) In yet another aspect, an artificial-reality system is provided andthe artificial-reality system includes means for performing the methoddescribed in any one of C1-C2 and D1-D2.

(E2) In another aspect, an artificial-reality system that includes ahead-mounted display and one or more cameras is provided. In someembodiments, the artificial-reality system is in communication with oneor more processors and memory storing one or more programs which, whenexecuted by the one or more processors, cause the artificial-realitysystem to perform the method described in any one of C1-C2 and D1-D2.

(E3) In still another aspect, a non-transitory computer-readable storagemedium is provided (e.g., as a memory device, such as external orinternal storage, that is in communication with an artificial-realitysystem). The non-transitory computer-readable storage medium storesexecutable instructions that, when executed by an artificial-realitysystem with one or more processors/cores, cause the artificial-realitysystem to perform the method described in any one of C1-C2 and D1-D2.

(F1) In accordance with some embodiments, another method is performed ata first wearable device, attached to a first appendage of a user, thatincludes one or more transducers. The method includes (i) receiving, bythe one or more transducers, a set of waves (e.g., signals) transmittedby a second wearable device attached to the user, wherein waves in theset of waves travel from the second wearable device to the firstwearable device through the first appendage of the user, (ii) afterreceiving the set of waves, determining first values for one or morewaveform characteristics of the set of waves, and (iii) identifying alocation of a touch gesture on the first appendage of the user based onthe first values for the one or more waveform characteristics of the setof waves. In some embodiments, the one or more waveform characteristicsincludes at least values for phase and amplitude.

(F2) In some embodiments of the method of F1, further includingreporting the location of the touch gesture to a computer system (e.g.,computer system 130, FIG. 1A).

(F3) In some embodiments of the method of any of F1-F2, the firstwearable device performs the method described in any one of B2-B15.

In accordance with some embodiments, a plurality of wearable device eachincludes one or more processors/cores and memory storing one or moreprograms configured to be executed by the one or more processors/cores.The one or more programs in each wearable devices includes instructionsfor performing one or more of the operations of the method describedabove. In accordance with some embodiments, a non-transitorycomputer-readable storage medium has stored therein instructions that,when executed by one or more processors/cores of a wearable device,cause the wearable device to perform some of the operations of themethod described above (e.g., operations of the first wearable device orthe second wearable device). In accordance with some embodiments, asystem includes a wearable device (or multiple wearable devices), ahead-mounted display (HMD), and a computer system to provide video/audiofeed to the HMD and instructions to the wearable device.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the various described embodiments,reference should be made to the Description of Embodiments below, inconjunction with the following drawings in which like reference numeralsrefer to corresponding parts throughout the figures and specification.

FIG. 1A is a block diagram illustrating an exemplary projection system,in accordance with various embodiments.

FIG. 1B is a block diagram illustrating an exemplary projection system,in accordance with various embodiments.

FIG. 2 is a block diagram illustrating an exemplary wearable device inaccordance with some embodiments.

FIG. 3 is a block diagram illustrating an exemplary computer system inaccordance with some embodiments.

FIG. 4A is an exemplary view of a wearable device on a user's wrist, inaccordance with some embodiments.

FIG. 4B is an exemplary cross-sectional view of a wearable device on auser's wrist, in accordance with some embodiments.

FIG. 5 is an exemplary cross-sectional view of a wearable device inaccordance with some embodiments.

FIG. 6A is an exemplary view of a wearable device on a user's wrist andon the user's head, in accordance with some embodiments.

FIG. 6B is an exemplary view of a wearable device on a user's wrist andon the user's finger, in accordance with some embodiments.

FIG. 6C is an exemplary view of a wearable device on a user's firstwrist and on the user's second wrist, in accordance with someembodiments.

FIG. 6D is an exemplary signal pathway between two wearable devices, inaccordance with some embodiments.

FIG. 7 is a flow diagram illustrating a method of projecting images ontoa user's body in accordance with some embodiments.

FIG. 8 is a flow diagram illustrating a method of confirming a touch ona user's body in accordance with some embodiments.

FIG. 9 is a high level flow diagram illustrating a method of detecting atouch on a user's body in accordance with some embodiments.

FIG. 10 is a flow diagram illustrating a method of confirming a touch ona user's body in accordance with some embodiments.

FIG. 11 illustrates an embodiment of an artificial reality device.

FIG. 12 illustrates an embodiment of an augmented reality headset and acorresponding neckband.

FIG. 13 illustrates an embodiment of a virtual reality headset.

DESCRIPTION OF EMBODIMENTS

Reference will now be made to embodiments, examples of which areillustrated in the accompanying drawings. In the following description,numerous specific details are set forth in order to provide anunderstanding of the various described embodiments. However, it will beapparent to one of ordinary skill in the art that the various describedembodiments may be practiced without these specific details. In otherinstances, well-known methods, procedures, components, circuits, andnetworks have not been described in detail so as not to unnecessarilyobscure aspects of the embodiments.

It will also be understood that, although the terms first, second, etc.are, in some instances, used herein to describe various elements, theseelements should not be limited by these terms. These terms are used onlyto distinguish one element from another. For example, a first wearabledevice could be termed a second wearable device, and, similarly, asecond wearable device could be termed a first wearable device, withoutdeparting from the scope of the various described embodiments. The firstwearable device and the second wearable device are both wearabledevices, but they are not the same wearable devices, unless specifiedotherwise.

The terminology used in the description of the various describedembodiments herein is for the purpose of describing particularembodiments only and is not intended to be limiting. As used in thedescription of the various described embodiments and the appendedclaims, the singular forms “a,” “an,” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. It will also be understood that the term “and/or” as usedherein refers to and encompasses any and all possible combinations ofone or more of the associated listed items. It will be furtherunderstood that the terms “includes,” “including,” “comprises,” and/or“comprising,” when used in this specification, specify the presence ofstated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups thereof.

As used herein, the term “if” is, optionally, construed to mean “when”or “upon” or “in response to determining” or “in response to detecting”or “in accordance with a determination that,” depending on the context.Similarly, the phrase “if it is determined” or “if [a stated conditionor event] is detected” is, optionally, construed to mean “upondetermining” or “in response to determining” or “upon detecting [thestated condition or event]” or “in response to detecting [the statedcondition or event]” or “in accordance with a determination that [astated condition or event] is detected,” depending on the context.

As used herein, the term “exemplary” is used in the sense of “serving asan example, instance, or illustration” and not in the sense of“representing the best of its kind.”

FIG. 1 is a block diagram illustrating a system 100, in accordance withvarious embodiments. While some example features are illustrated,various other features have not been illustrated for the sake of brevityand so as not to obscure pertinent aspects of the example embodimentsdisclosed herein. To that end, as anon-limiting example, the system 100includes wearable devices 102 a, 102 b, which are used in conjunctionwith a computer system 130 (e.g., a host system or a host computer). Insome embodiments, the system 100 provides the functionality of a virtualreality device with image projection, an augmented reality device withimage projection, a combination thereof, or provides some otherfunctionality. The system 100 is described in greater detail below withreference FIGS. 11-13.

An example wearable device 102 (e.g., wearable device 102 a) includes,for example, one or more processors/cores 104 (referred to henceforth as“processors”), a memory 106, one or more transducer arrays 110, one ormore communications components 112, projector(s) 115, and/or one or moresensors 114. In some embodiments, these components are interconnected byway of a communications bus 108. References to these components of thewearable device 102 cover embodiments in which one or more of thesecomponents (and combinations thereof) are included. In some embodiments,the one or more sensors 114 are part of the one or more transducerarrays 110 (e.g., transducers in the transducer arrays 110 also performthe functions of the one or more sensors 114, discussed in furtherdetail below). For example, one or more transducers in the transducerarray 110 may be electroacoustic transducers configured to detectacoustic waves (e.g., ultrasonic waves).

Another example wearable device 102 (e.g., wearable device 102 b)includes, for example, one or more processors/cores 104 (referred tohenceforth as “processors”), a memory 106, one or more transducer arrays110, one or more communications components 112, camera(s) 118, and/orone or more sensors 114. In some embodiments, these components areinterconnected by way of a communications bus 108. References to thesecomponents of the wearable device 102 cover embodiments in which one ormore of these components (and combinations thereof) are included. Insome embodiments, the one or more sensors 114 are part of the one ormore transducer arrays 110 (e.g., transducers in the transducer arrays110 also perform the functions of the one or more sensors 114, discussedin further detail below). For example, one or more transducers in thetransducer array 110 may be electroacoustic transducers configured todetect acoustic waves (e.g., ultrasonic waves).

In some embodiments, a single processor 104 (e.g., processor 104 of thewearable device 102 a) executes software modules for controllingmultiple wearable devices 102 (e.g., wearable devices 102 b . . . 102n). In some embodiments, a single wearable device 102 (e.g., wearabledevice 102 a) includes multiple processors 104, such as one or morewearable device processors (configured to, e.g., generate an image forprojection), one or more communications component processors (configuredto, e.g., control communications transmitted by communications component112 and/or receive communications by way of communications component112) and/or one or more sensor processors (configured to, e.g., controloperation of sensor 114 and/or receive output from sensor 114).

In some embodiments, the wearable device 102 is configured to projectimage(s) 602 (as shown in FIG. 6A) via the projector(s) 115 withinprojection unit 412 (shown in FIG. 4A). In such embodiments, thewearable device 102 is configured to generate and project images (e.g.,a keyboard or the like) onto the user's own appendage using, e.g., oneor more of the one or more projectors 115. The AR system 1100 (FIG. 11)shows an example wearable device that can project images (at least insome embodiments).

In some other embodiments, the wearable device 102 does not projectimages and instead the computer system 130 (and the head-mounted display140) is (are) responsible for projecting images onto the user's ownappendage. Alternatively, in some embodiments, the computer system 130(and the head-mounted display 140) uses augmented reality so that theuser perceives images on his or her own appendage, but nothing isactually projected. AR system 1200 (FIG. 12) and VR system 1300 (FIG.13) can be used to project/display images onto the user or areas aroundthe user.

In some embodiments, the transducers in a respective transducer array110 are miniature piezoelectric actuators/devices, vibrotactileactuators, or the like. In some embodiments, the transducers in arespective transducer array 110 are single or multipole voice coilmotors, or the like. Each transducer array 110 is configured to generateand transmit signals 116 in response to being activated by the wearabledevice (e.g., via processors 104 or some other controller included inthe wearable device 102). In some embodiments, the signals 116 aremechanical waves (e.g., sound waves, ultrasonic waves, or various othermechanical waves). A mechanical wave is an oscillation of matter thattransfers energy through a medium. As discussed herein, the “medium” isthe wearer's skin, flesh, bone, blood vessels, etc. It is noted that anydevice capable of producing mechanical waves (or alternating currentsignals) can be used as a transducer in the disclosed wearable device102. It is also noted that signals (e.g., waves) that propagate throughthe medium (e.g., the user's flesh) are said herein to “couple” to themedium or “couple into” the medium.

In some embodiments, the wearable device 102 (e.g., wearable device 102a, 102 b) is a receiver and transmitter of one or more signals. Forexample, in addition to transmitting signals (e.g., mechanical waves),as described above, the wearable device 102 is also configured toreceive (e.g., detect, sense) signals transmitted by itself or byanother wearable device 102. To illustrate, a first wearable device 102a may transmit a plurality of signals through a medium, such as thewearer's body, and a second wearable device 102 b (attached to the samewearer) may receive at least some of the signals transmitted by thefirst wearable device 102 a through the medium. Furthermore, a wearabledevice 102 receiving transmitted signals may use the received signals todetermine that a user contacted a particular part of his or her body. Toillustrate, the second wearable device 102 b may initially receivesignals transmitted by the first wearable device 102 a through themedium that have a first set of parameters (e.g., values of phase,amplitude, frequency, etc.). The second wearable device 102 b may usethese initial signals to form a normalized baseline. Thereafter, thewearer of the first and second wearable devices 102 may contact (e.g.,touch) a region of her body (e.g., forearm) through which thetransmitted signals are travelling. By touching her forearm for example,the wearer alters the signals travelling through her forearm, and inturn the first set of parameters associated with the signals (e.g.,values of one or more of phase, amplitude, frequency, etc. may change).Importantly, the second wearable device 102 b then receives (e.g.,detects, senses) these altered signals and can subsequently determinethat the user contacted a particular part of her body, e.g., herforearm. The second wearable device 102 b may further determine that theuser contacted a specific part of her forearm (e.g., a change in thephase value by a certain amount from the normalized baseline mayindicate that a specific part of her forearm was touched).

The computer system 130 is a computing device that executes virtualreality applications and/or augmented reality applications to processinput data from the sensors 145 on the head-mounted display 140 and thesensors 114 on the wearable device 102. The computer system 130 providesoutput data to at least (i) the electronic display 144 on thehead-mounted display 140 and (ii) the wearable device 102 (e.g.,processors 104 of the haptic device 102, FIG. 2A). An exemplary computersystem 130, for example, includes one or more processor(s)/core(s) 132,memory 134, one or more communications components 136, and/or one ormore cameras 139. In some embodiments, these components areinterconnected by way of a communications bus 138. References to thesecomponents of the computer system 130 cover embodiments in which one ormore of these components (and combinations thereof) are included.

In some embodiments, the computer system 130 is a standalone device thatis coupled to a head-mounted display 140. For example, the computersystem 130 has processor(s)/core(s) 132 for controlling one or morefunctions of the computer system 130 and the head-mounted display 140has processor(s)/core(s) 141 for controlling one or more functions ofthe head-mounted display 140. Alternatively, in some embodiments, thehead-mounted display 140 is a component of computer system 130. Forexample, the processor(s) 132 controls functions of the computer system130 and the head-mounted display 140. In addition, in some embodiments,the head-mounted display 140 includes the processor(s) 141 thatcommunicate with the processor(s) 132 of the computer system 130. Insome embodiments, communications between the computer system 130 and thehead-mounted display 140 occur via a wired (or wireless) connectionbetween communications bus 138 and communications bus 146. In someembodiments, the computer system 130 and the head-mounted display 140share a single communications bus. It is noted that in some instancesthe head-mounted display 140 is separate from the computer system 130(as shown in FIG. 11).

The computer system 130 may be any suitable computer device, such as alaptop computer, a tablet device, a netbook, a personal digitalassistant, a mobile phone, a smart phone, a virtual reality device(e.g., a virtual reality (VR) device, an augmented reality (AR) device,or the like), a gaming device, a computer server, or any other computingdevice. The computer system 130 is sometimes called a host or a hostsystem. In some embodiments, the computer system 130 includes other userinterface components such as a keyboard, a touch-screen display, amouse, a track-pad, and/or any number of supplemental I/O devices to addfunctionality to computer system 130.

In some embodiments, one or more cameras 139 of the computer system 130are used to facilitate virtual reality and/or augmented reality.Moreover, in some embodiments, the one or more cameras 139 also act asprojectors to display the virtual and/or augmented images (or in someembodiments the computer system includes one or more distinctprojectors). In some embodiments, the computer system 130 providesimages captured by the one or more cameras 139 to the display 144 of thehead-mounted display 140, and the display 144 in turn displays theprovided images. In some embodiments, the processors 141 of thehead-mounted display 140 process the provided images. It is noted thatin some embodiments, one or more of the cameras 139 are part of thehead-mounted display 140.

The head-mounted display 140 presents media to a user. Examples of mediapresented by the head-mounted display 140 include images, video, audio,or some combination thereof. In some embodiments, audio is presented viaan external device (e.g., speakers and/or headphones) that receivesaudio information from the head-mounted display 140, the computer system130, or both, and presents audio data based on the audio information.The displayed images may be in virtual reality, augmented reality, ormixed reality. An exemplary head-mounted display 140, for example,includes one or more processor(s)/core(s) 141, a memory 142, and/or oneor more displays 144. In some embodiments, these components areinterconnected by way of a communications bus 146. References to thesecomponents of the head-mounted display 140 cover embodiments in whichone or more of these components (and combinations thereof) are included.It is noted that in some embodiments, the head-mounted display 140includes one or more sensors 145. Alternatively, in some embodiments,the one or more sensors 145 are part of the computer system 130. FIGS.12 and 13 illustrate additional examples (e.g., AR system 1200 and VRsystem 1300) of the head-mounted display 140.

The electronic display 144 displays images to the user in accordancewith data received from the computer system 130. In various embodiments,the electronic display 144 may comprise a single electronic display ormultiple electronic displays (e.g., one display for each eye of a user).

The sensors 145 include one or more hardware devices that detect spatialand motion information about the head-mounted display 140. Spatial andmotion information can include information about the position,orientation, velocity, rotation, and acceleration of the head-mounteddisplay 140. For example, the sensors 145 may include one or moreinertial measurement units (IMUs) that detect rotation of the user'shead while the user is wearing the head-mounted display 140. Thisrotation information can then be used (e.g., by the computer system 130)to adjust the images displayed on the electronic display 144. In someembodiments, each IMU includes one or more gyroscopes, accelerometers,and/or magnetometers to collect the spatial and motion information. Insome embodiments, the sensors 145 include one or more cameras positionedon the head-mounted display 140.

In some embodiments, the transducer array 110 of the wearable device 102may include one or more transducers configured to generate and/orreceive signals. Integrated circuits (not shown) of the wearable device102, such as a controller circuit and/or signal generator (e.g.,waveform generator), may control the behavior of the transducers (e.g.,controller 412, FIG. 4A).

The communications component 112 of the wearable device 102 may includea communications component antenna for communicating with the computersystem 130. Moreover, the communications component 136 may include acomplementary communications component antenna that communicates withthe communications component 112. The respective communicationcomponents are discussed in further detail below with reference to FIGS.2 and 3.

In some embodiments, data contained within communication signals is usedby the wearable device 102 for selecting and/or generating projectionimages. In some embodiments, the data contained within the communicationsignals alerts the computer system 130 that the wearable device 102 isready for use. As will be described in more detail below, the computersystem 130 sends instructions to the wearable device 102, and inresponse to receiving the instructions, the wearable device generatesprojection images 602 that are displayed on an appendage of the user ofthe wearable device 102. Alternatively or in addition, in someembodiments, the wearable device 102 sends signals to the computerdevice 130 that include information indicating a location of a touch onthe user's body (or a position of an appendage with respect to aposition of the wearable device). As explained above, a wearable devicereceiving signals transmitted by another wearable device is able todetermine, based on changes of signal parameters caused by the touch, alocation of the touch on the wearer's body. As one example, a keyboard(or some other user interface) may be projected or perceived to beprojected onto the user's forearm, and the wearable device maydetermine, based on changes of signal parameters caused by the touch,that the user is intending to interact with a first affordance of thekeyboard. In this way, the system 100 provides a novel way ofdetermining where (and/or whether) a person contacts his or her skin(e.g., in combination with or separate from other video-based means formaking this determination). This is particularly useful when augmentedreality is being used, and actual images are not in fact projected ontothe user's body. In another example, the wearable device may determine,based on changes of signal parameters, that the user touched herforearm. Information related to the touch may then be sent to thecomputer device 130, and used by the computer device 130 to confirm thata touch occurred on the forearm.

Non-limiting examples of sensors 114 and/or sensors 145 include, e.g.,infrared, pyroelectric, ultrasonic, microphone, laser, optical, Doppler,gyro, accelerometer, resonant LC sensors, capacitive sensors, acousticsensors, and/or inductive sensors. In some embodiments, sensors 114and/or sensors 145 are configured to gather data that is used todetermine a hand posture of a user of the wearable device and/or animpedance of the medium. Examples of sensor data output by these sensorsinclude: body temperature data, infrared range-finder data, motion data,activity recognition data, silhouette detection and recognition data,gesture data, heart rate data, and other wearable device data (e.g.,biometric readings and output, accelerometer data). In some embodiments,the transducers themselves serve as sensors.

FIG. 1B is a block diagram illustrating an embodiment of the system 100,in accordance with various embodiments. The system 100 includes wearabledevices 102 a, 102 b, and 102 c which are used in conjunction with acomputer system 130 (e.g., a host system or a host computer). Wearabledevice 102 c may be an additional device worn by the user to be used inconjunction with wearable devices 102 a and 102 b. For example, thewearable device 102 c may be a ring which may be used to determine alocation of a touch gesture. In another example, the wearable device 102a and wearable device 102 c may be distinct wristbands to be worn oneach wrist of the user. In some embodiments, the system 100 provides thefunctionality of a virtual-reality device with image projection, anaugmented reality device with image projection, a combination thereof,or provides some other functionality. In some embodiments, the wearabledevice 102 c may include all or some of the features embodied in thewearable devices 102 a, 102 b.

FIG. 2 is a block diagram illustrating a representative wearable device102 in accordance with some embodiments. In some embodiments, thewearable device 102 includes one or more processing units (e.g., CPUs,microprocessors, and the like) 104, one or more communication components112, memory 106, one or more transducer arrays 110, one or moreprojectors 115, one or more cameras 118, and one or more communicationbuses 108 for interconnecting these components (sometimes called achipset). In some embodiments, the wearable device 102 includes one ormore sensors 114 as described above with reference to FIG. 1. In someembodiments (not shown), the wearable device 102 includes one or moreoutput devices such as one or more indicator lights, sound cards,speakers, displays for displaying textual information and error codes,etc.

Transducers in a respective transducer array 110 generate signals 116(FIG. 1). In some embodiments, the transducers may include, e.g.,hardware capable of generating the signals 116 (e.g., electromagneticwaves, soundwaves, ultrasound waves, etc.). For example, each transducercan convert electrical signals into ultrasound waves. The transducersmay be miniature piezoelectric transducers, capacitive transducers,single or multipole voice coil motors, and/or any other suitable devicefor creation of signals. Additionally, in some embodiments, thetransducers can also receive signals (e.g., if the transducer cangenerate sound waves, it can also receive sound waves). Continuing, insome embodiments, the transducers may also be any of the sensors 114described above with reference to FIG. 1. In some embodiments, a firstwearable device 102 a includes first transducers (e.g., transducers forreceiving, sensing, detecting, etc.) while a second wearable 102 bincludes second transducers (e.g., transducers for generates signals tobe sensed by the first transducers) distinct from the first transducers.

The communication component(s) 112 enable communication between thewearable device 102 and one or more communication networks. In someembodiments, the communication component(s) 112 include, e.g., hardwarecapable of data communications using any of a variety of wirelessprotocols (e.g., IEEE 802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave,Bluetooth Smart, ISA100.11a, WirelessHART, MiWi, etc.), wired protocols(e.g., Ethernet, HomePlug, etc.), and/or any other suitablecommunication protocol, including communication protocols not yetdeveloped as of the filing date of this document.

The memory 106 includes high-speed random access memory, such as DRAM,SRAM, DDR SRAM, or other random access solid state memory devices; and,optionally, includes non-volatile memory, such as one or more magneticdisk storage devices, one or more optical disk storage devices, one ormore flash memory devices, or one or more other non-volatile solid statestorage devices. The memory 106, or alternatively the non-volatilememory within memory 106, includes a non-transitory computer-readablestorage medium. In some embodiments, the memory 106, or thenon-transitory computer-readable storage medium of the memory 106,stores the following programs, modules, and data structures, or a subsetor superset thereof:

-   -   operating logic 216 including procedures for handling various        basic system services and for performing hardware dependent        tasks;    -   communication module 218 for coupling to and/or communicating        with remote devices (e.g., computer system 130, other wearable        devices, etc.) in conjunction with communication component(s)        112;    -   sensor module 220 for obtaining and processing sensor data        (e.g., in conjunction with sensor(s) 114 and/or transducer        arrays 110) to, for example, determine an orientation of the        wearable device 102 and sensing signals generated by one or more        transducers (among other purposes such as determining hand pose        of the user of the wearable device);    -   signal generating module 222 for generating and transmitting        (e.g., in conjunction with transducers(s) 110) signals. In some        embodiments, the module 222 also includes or is associated with        a data generation module 223 that is used to generate data        corresponding to the received portion of the transmitted signals        (e.g., data for a candidate touch event);    -   database 224, including but not limited to:        -   sensor information 226 for storing and managing data            received, detected, and/or transmitted by one or more            sensors (e.g., sensors 114, one or more remote sensors,            and/or transducer arrays 110);        -   device settings 228 for storing operational settings for the            wearable device 102 and/or one or more remote devices (e.g.,            selected characteristics/parameters values for the signals);            and        -   communication protocol information 230 for storing and            managing protocol information for one or more protocols            (e.g., custom or standard wireless protocols, such as            ZigBee, Z-Wave, etc., and/or custom or standard wired            protocols, such as Ethernet);    -   projection module 232 for projecting one or more images onto an        appendage of the wearer or user of the wearable device;    -   tactile gesture detection module 234 for detecting a touch        gesture on the one or more projected images projected via        projector 115, including but not limited to:        -   tactile location information 236 for detecting a location of            the touch gesture on the one or more projected images; and    -   computer vision gesture detection module for detecting a touch        gesture on the one or more projected images detected via camera        118, including but not limited to:        -   computer vision location information 240 for detecting a            location of the touch gesture on the one or more projected            images using computer vision analysis.

In some embodiments, the tactile gesture detection module 234 uses aknown impedance map of the user's body, capacitive couplingtechnologies, signal processing techniques, and/or acoustic wavecoupling (e.g., ultrasound waves) when determining a location of thetouch gesture. In some embodiments, the tactile gesture detection module234 communicates with the sensor module 220 to determine a location ofthe touch gesture on the user's body (e.g., based on the sensor dataobtained by the sensor module 220, the tactile gesture detection module234 can determine a location of the touch gesture). In some embodiments,the tactile gesture detection module 234 and/or the computer visiongesture detection module 238 is (are) located at the computer system130.

In some embodiments, the location information 236, 240 is determinedusing computer vision technologies and/or non-optical imaging techniquesusing capacitance, magnetism, and millimeter wave technologies and/oracoustic wave coupling (e.g., ultrasound waves).

In some embodiments (not shown), the wearable device 102 includes alocation detection device, such as a GNSS (e.g., GPS, GLONASS, etc.) orother geo-location receiver, for determining the location of thewearable device 102. Further, in some embodiments, the wearable device102 includes location detection module (e.g., a GPS, Wi-Fi, magnetic, orhybrid positioning module) for determining the location of the wearabledevice 102 (e.g., using the location detection device) and providingthis location information to the host system 130.

In some embodiments (not shown), the wearable device 102 includes aunique identifier stored in database 224. In some embodiments, thewearable device 102 sends the unique identifier to the host system 130to identify itself to the host system 130. This is particularly usefulwhen multiple wearable devices are being concurrently used.

Each of the above-identified elements (e.g., modules stored in memory106 of the wearable device 102) is optionally stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing the function(s) described above. The aboveidentified modules or programs (e.g., sets of instructions) need not beimplemented as separate software programs, procedures, or modules, andthus various subsets of these modules are optionally combined orotherwise rearranged in various embodiments. In some embodiments, thememory 106, optionally, stores a subset of the modules and datastructures identified above. Furthermore, the memory 106, optionally,stores additional modules and data structures not described above.

FIG. 3 is a block diagram illustrating a representative computer system130 in accordance with some embodiments. In some embodiments, thecomputer system 130 includes one or more processing units/cores (e.g.,CPUs, GPUs, microprocessors, and the like) 132, one or morecommunication components 136, memory 134, one or more cameras 139, andone or more communication buses 308 for interconnecting these components(sometimes called a chipset). In some embodiments, the computer system130 includes a head-mounted display interface 305 for connecting thecomputer system 130 with the head-mounted display 140. As discussedabove in FIG. 1, in some embodiments, the computer system 130 and thehead-mounted display 140 are together in a single device, whereas inother embodiments the computer system 130 and the head-mounted display140 are separate from one another.

Although not shown, in some embodiments, the computer system (and/or thehead-mounted display 140) includes one or more sensors 145 (as discussedabove with reference to FIG. 1) and/or one or more instances of thetransducer arrays 110.

The communication component(s) 136 enable communication between thecomputer system 130 and one or more communication networks. In someembodiments, the communication component(s) 136 include, e.g., hardwarecapable of data communications using any of a variety of custom orstandard wireless protocols (e.g., IEEE 802.15.4, Wi-Fi, ZigBee,6LoWPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.11a, WirelessHART,MiWi, etc.), custom or standard wired protocols (e.g., Ethernet,HomePlug, etc.), and/or any other suitable communication protocol,including communication protocols not yet developed as of the filingdate of this document.

The memory 134 includes high-speed random access memory, such as DRAM,SRAM, DDR SRAM, or other random access solid state memory devices; and,optionally, includes non-volatile memory, such as one or more magneticdisk storage devices, one or more optical disk storage devices, one ormore flash memory devices, or one or more other non-volatile solid statestorage devices. The memory 134, or alternatively the non-volatilememory within memory 134, includes a non-transitory computer-readablestorage medium. In some embodiments, the memory 134, or thenon-transitory computer-readable storage medium of the memory 134,stores the following programs, modules, and data structures, or a subsetor superset thereof:

-   -   operating logic 316 including procedures for handling various        basic system services and for performing hardware dependent        tasks;    -   communication module 318 for coupling to and/or communicating        with remote devices (e.g., wearable devices 102 a-102-n, a        remote server (not shown), etc.) in conjunction with        communication component(s) 136;    -   virtual-reality generation module 320 that is used for        generating virtual-reality images and sending corresponding        video and audio data to the HMD 140 (in some embodiments, the        virtual-reality generation module 320 is an augmented-reality        generation module 320 (or the memory 134 includes a distinct        augmented-reality generation module) that is used for generating        augmented-reality images and projecting those images in        conjunction with the camera(s) 139 and the HMD 140);    -   instruction module 322 that is used for generating an        instruction that, when sent to the wearable device 102 (e.g.,        using the communications component 136), causes the wearable        device 102 to activate two or more transducers;    -   display module 324 that is used for displaying virtual-reality        images and/or augmented-reality images in conjunction with the        head-mounted display 140 and/or the camera(s) 139;    -   computer vision gesture detection module 338 for detecting a        touch gesture detected via camera 139, including but not limited        to:        -   computer vision location information 340 for detecting a            location of the touch gesture using computer vision            analysis.    -   database 326, including but not limited to:        -   display information 328 for storing (and generating)            virtual-reality images and/or augmented-reality images            (e.g., visual data);        -   haptics information 330 for storing (and generating) haptics            information that corresponds to displayed virtual-reality            images and environments and/or augmented-reality images and            environments;        -   communication protocol information 332 for storing and            managing protocol information for one or more protocols            (e.g., custom or standard wireless protocols, such as            ZigBee, Z-Wave, etc., and/or custom or standard wired            protocols, such as Ethernet); and        -   mapping data 334 for storing and managing mapping data            (e.g., mapping one or more wearable devices 102 on a user).

In the example shown in FIG. 3, the computer system 130 further includesvirtual-reality (and/or augmented-reality) applications 336. In someembodiments, the virtual-reality applications 336 are implemented assoftware modules that are stored on the storage device and executed bythe processor. Each virtual-reality application 336 is a group ofinstructions that, when executed by a processor, generates virtual oraugmented reality content for presentation to the user. Avirtual-reality application 336 may generate virtual/augmented-realitycontent in response to inputs received from the user via movement of thehead-mounted display 140 or the wearable device 102. Examples ofvirtual-reality applications 336 include gaming applications,conferencing applications, and video playback applications.

The virtual-reality generation module 320 is a software module thatallows virtual-reality applications 336 to operate in conjunction withthe head-mounted display 140 and the wearable device 102. Thevirtual-reality generation module 320 may receive information from thesensors 145 on the head-mounted display 140 and may, in turn provide theinformation to a virtual-reality application 336. Based on the receivedinformation, the virtual-reality generation module 320 determines mediacontent to provide to the head-mounted display 140 for presentation tothe user via the electronic display 144. For example, if thevirtual-reality generation module 320 receives information from thesensors 145 on the head-mounted display 140 indicating that the user haslooked to the left, the virtual-reality generation module 320 generatescontent for the head-mounted display 140 that mirrors the user'smovement in a virtual/augmented environment. An example VR system 1300is provided in FIG. 13.

Similarly, in some embodiments, the virtual-reality generation module320 receives information from the sensors 114 on the wearable device 102and provides the information to a virtual-reality application 336. Theapplication 336 can use the information to perform an action within thevirtual/augmented world of the application 336. For example, if thevirtual-reality generation module 320 receives information from thesensors 114 that the user has raised his hand, a simulated hand (e.g.,the user's avatar) in the virtual-reality application 336 lifts to acorresponding height. As noted above, the information received by thevirtual-reality generation module 320 can also include information fromthe head-mounted display 140. For example, cameras 139 on thehead-mounted display 140 may capture movements of the user (e.g.,movement of the user's arm), and the application 336 can use thisadditional information to perform the action within thevirtual/augmented world of the application 336.

To further illustrate with an augmented reality example, if theaugment-reality generation module 320 receives information from thesensors 114 that the user has rotated his forearm while, in augmentedreality, a user interface (e.g., a keypad) is displayed on the user'sforearm, the augmented-reality generation module 320 generates contentfor the head-mounted display 140 that mirrors the user's movement in theaugmented environment (e.g., the user interface rotates in accordancewith the rotation of the user's forearm). An example AR system 1200 isprovided in FIG. 12.

In some embodiments, the computer system 130 receives sensor data fromthe wearable device 102 and the computer system 130 includes a module todetermine a touch location associated with the sensor data. In someembodiments, the computer system 130 determines a touch location (e.g.,using the computer vision gesture detection module 338) based on sensordata from the wearable device 102 and image data captured by the one ormore camera 139. In this way, a majority of the processing is offloadedfrom the wearable device 102 to the computer system 130, which may haveincreased processing abilities.

Each of the above identified elements (e.g., modules stored in memory134 of the computer system 130) is optionally stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing the function(s) described above. The aboveidentified modules or programs (e.g., sets of instructions) need not beimplemented as separate software programs, procedures, or modules, andthus various subsets of these modules are optionally combined orotherwise rearranged in various embodiments. In some embodiments, thememory 134, optionally, stores a subset of the modules and datastructures identified above.

FIG. 4A is an example view of the wearable device 102 in accordance withsome embodiments. The example view shows the user's hand 408, user'swrist 404, user's arm 406, and the wearable device 102 on the user's arm406. Such an arrangement is merely one possible arrangement, and oneskilled in the art will appreciate that the discussion herein is notlimited to the arrangement shown in FIG. 4A. Additionally, the wearabledevice 102 shown in FIG. 4A is shown oversized for ease of illustration.In practice, a size of the wearable device 102 can be reduced, ifdesired, so that the wearable device 102 has a size similar to a smartwatch or fitness tracker.

The wearable device 102 includes a wearable structure 402 that may be aflexible mechanical substrate such as a plastic (e.g., polyethylene orpolypropylene), rubber, nylon, synthetic, polymer, etc. In someembodiments, the wearable structure 402 is configured to be worn aroundat least a portion of a user's wrist or arm 404/406 (and various otherbody parts). The wearable device 102 includes a transducer array 110,including a plurality of transducers 410 arranged at different locationson the wearable structure 402. The transducers 410 can be arranged in apattern along an inner surface of the wearable structure 402 facing thearm 406 such that the transducers 410 contact the user's skin. Inanother example, the transducers can be arranged in a radial patternalong an inner perimeter of the wearable structure 502 (FIG. 5).

In some embodiments, a respective transducer 410 is configured togenerate signals (e.g., waves 116, FIG. 1) in response to receiving oneor more control signals from a controller (not shown). The one or morecontrol signals instruct one or more transducers 410 in the transducerarray 110 to send signals (e.g., ultrasonic waves) into/through theuser's body (e.g., wrist or arm). The signals transmitted by the one ormore transducers 410 are to travel (e.g., propagate, radiate) away fromthe wearable structure 402 through the user's body. For example, thesignals may travel from the user's arm 406, through the user's wrist404, to the user's hand 408 and fingers. In addition, the signals maytravel up the user's arm and eventually travel throughout the user'sbody.

In some embodiments, the wearable structure 402 includes a projectionunit 412 (e.g., projector 115, FIG. 1A) that projects images onto anappendage of a user. In some embodiments, the wearable structure 402includes a memory (e.g., memory 106, FIG. 1) that stores images to bedisplayed. For example, the stored images may represent a user interfacehaving keyboard with multiple affordances (various other “touch”interfaces could also be projected onto the user's appendage). In someembodiments, the controller 412 generates a control signal (or multiplesignals) based on an instruction from a host system (e.g., computersystem 130, FIG. 1). In such embodiments, the wearable device 102 isplaced on a user's arm 406 (or various other locations) to projectimages onto the forearm 406 of the user.

Alternatively, in some embodiments, the wearable device 102 does notproject images but instead, through the computer system 130 and thehead-mounted display 140, images are perceived on the user's arm (orother body part) through augmented reality. In such embodiments, thewearable device 102 is configured to sense interaction with the user'sbody. Put another way, the wearable device 102 is used to track avirtual image virtualized by another device (e.g., head-mounted display140), as if the image were projected onto the user's arm. For example, acamera and/or projector of the other device (e.g., headset, glasses,head-mounted display 140) may project an image onto its own lens (e.g.,display 144, FIG. 1). Augmented reality technologies (e.g., AR system1200, FIG. 12) may implement such an embodiment.

In some embodiments, the other device tracks the user's arm 406 andadjusts the image (e.g., in augmented reality) in accordance withmovements of the user's arm 406. For example, cameras 139 may be used totrack the user's arm 406.

In some embodiments, the wearable devices 102 are worn in conjunctionwith one another. In some embodiments, the user may wear a singlewearable device.

In some embodiments, the wearable devices 102 may be configured tointeract with a computer system (e.g., computer system 130) that doesnot include a visual output. For example, wearable device 102 may beused to control settings on a phone while the user is on a call andunable to access the screen.

In some embodiments, the transducer array 110 includes transducers 410designed to make contact with human skin. A contact area having aconductive agent 462 and padding may be used on the wearable device 102behind each transducer to improve subject comfort and reduce contactimpedances (e.g., as shown in FIG. 5). The conductive agent between thetransducer and skin may be a “wet” connection using a conductive gel,which may consist of propylene glycol and NaCl, or a “dry” connection,such as a thin layer of conductive polymer (e.g., carbon-doped PDMS).

FIG. 4B is an example cross sectional view of the wearable device 120taken along the X-Y line shown in FIG. 4A, in accordance with someembodiments. The cross sectional view shows the user's arm 406 and atendon 455 within the user's arm 406. In this particular example, thetransducers 410 do not fully wrap around the wrist (e.g., transducers410-A-410-D are disposed on one side of the user's arm 406).

One or more of the transducers 410-A-410-D can generate signals (e.g.,waves 454-A and 454-B) in the user's arm 406. The generated signals454-A and 454-B may extend into the user's body (e.g., extend into theepidermis, the dermis, the muscles, the tendons, the ligaments, thebones, etc.). In some embodiments, each transducer 410 varies one ormore of a time period of the signal, an amplitude of the signal, and aphase of the signal when generating the signals.

To provide some content, the generated signals 454-A, 454-B, or aportion of the signals 454-A, 454-B, are reflected by the tendon 455and/or a portion of the wearable structure 402. As a result, thereflected signals 456-A, 456-B are received by the transducers 410-A and410-D. In some instances, the same transducers that generate the signalsdo not receive the signals. While not shown in FIG. 4B, one or moresignals transmitted by the wearable device 400 may travel through theuser's appendage and may be received (e.g., sensed) by a differentwearable device attached to the user.

In some embodiments, the transducers 410 transmit signals into theuser's body in a staggered manner, where different subsets of thetransducers transmit signals at different times. In some embodiments,the remaining transducers may be used to measure the altered signalsthat they receive. This procedure may then be repeated for multiplestimulation patterns defining an order of transducers (e.g., pairs oftransducers) selected to emit the signals.

FIG. 5 is an exemplary cross-sectional view of a wearable device inaccordance with some embodiments. The wearable device 500 (e.g.,wearable device 102, FIG. 1A, 1B) includes a wearable structure 502. Thewearable structure 502 wraps around the part of the user's body. Thewearable device 500 further includes a transducer array 110 having aplurality of transducers 410 positioned along an inner perimeter of thewearable structure 502. The transducers 410 in this example are radiallyspaced, such that the transducers 410 wrap around the wearable structure502 and form a substantially contiguous circle of transducers. In suchan arrangement, the wearable device 500 is able to produce signals 116in a 360-degree fashion. In some embodiments, the wearable structure 502separates the transducers 410 from the user's skin. Alternatively, insome embodiments (not shown), the transducers 410 are in direct contactwith the user's skin (a conductive agent may also be included). In someembodiments, the wearable structure 502 includes one or more projectors506 (e.g., projector 115, FIG. 1A-1B).

The wearable device 500 is configured to be attached to a part of auser's body. For example, the wearable device 500 is configured to beattached to a wrist, forearm, ankle, bicep, calf, thigh, scalp, and/orvarious other parts of the user's body. In some embodiments, thewearable device 500 is a rigid or semi-rigid structure. Alternatively,in some embodiments, the wearable device 500 is a flexible structure.Although the wearable device 500 is shown as a continuous circle, thewearable device 500 may break apart to be attached to the user's body(e.g., in a similar fashion to a watch).

FIG. 6A is an exemplary view of a wearable device on a user's wrist andon the user's head in accordance with some embodiments. In someembodiments, the wearable device 102 a is worn on the wrist of theuser's arm and the wearable device 102 b is worn on the head of theuser. In some embodiments, the wearable device 102 a uses one or moreprojectors 115 to project an image 602 onto the arm of the user. In someembodiments, the wearable device 102 b includes a camera 118 used forcomputer vision. In some embodiments, computer vision is used to detecta general position of the wearable device 102 a and/or the generalposition of the user's limb (e.g., hand, arm, fingers). In someembodiments, transducers 410 of the wearable device 102 a may determinethe magnitude and duration of a touch gesture (e.g., determine whether auser's finger is hovering over the skin, the user's finger is makingdirect contact with the skin).

In some embodiments, the wearable device 102 a does not project imagesand instead the wearable device 102 b is responsible for projectingimage 602 (e.g., a user interface) onto the arm of the user.Alternatively, in some embodiments, the wearable device 102 b usesaugmented reality so that the user perceives the image 602 on his or herarm, but nothing is actually projected. It is noted that the wearabledevice 102 b may be replaced with the computer system 130 (and thehead-mounted display 140). Examples of the computer system 130 and thehead-mounted display 140 are provided in FIGS. 12 and 13. AR system 1200(FIG. 12) and VR system 1300 (FIG. 13) can be used to project/displayimages onto the user or areas around the user.

FIG. 6B is an exemplary view of wearable devices on a user's wrist andon the user's finger, in accordance with some embodiments. In someembodiments, a first wearable device 102 a is worn on the wrist of theuser's arm and a second wearable device 102 c is worn on a finger on theother arm of the user. In some embodiments, the first wearable device102 a uses one or more projectors 115 to project the image 602 onto theuser's arm. Furthermore, the first wearable device 102 a may use the oneor more projectors 115 or one or more cameras 118 to detect touchgesture with respect to the projected image 602. The touch gesture 804may be one or more of a tap gesture, a swipe gesture, a pinch gesture, apull gesture, a twist gesture, etc. on the user's body. In someembodiments, as noted above, the image 602 is not projected onto theuse's arm by the wearable device 102 a. Instead, the image 602 isperceived in augmented reality. For example, a wearable device (e.g.,head-mounted display 140) may display the image 602 onto one or more ofthe displays 144. Furthermore, the computer system 130 is configured toadjust the display of the image 602 based on detected movement of theuser's arm (discussed above with reference to FIG. 3).

FIG. 6C is an exemplary view of wearable devices on a user's first wristand second wrist, in accordance with some embodiments. The arrangementof wearable devices 102 shown in FIG. 6C is used to detect a touchlocation 604 on the user's body. In some embodiments, a camera is usedto detect the touch location 604. Alternatively or in addition, in someembodiments, detected changes in signal parameters are used to detectthe touch location 604 or, more broadly, that a touch occurred.

As shown, a first wearable device 102 a is worn on the left wrist of theuser's left arm and a second wearable device 102 c is worn on the rightwrist of the user's right arm. In some embodiments, the first and secondwearable devices 102 are identical. For example, the first and secondwearable devices 102 include the same arrangement and types oftransducers 410. Alternatively, in some embodiments, the first andsecond wearable devices 102 differ in some way. For example, transducersof the first wearable device 102 a may differ from the transducers ofthe second wearable device 102 c. The first wearable device 102 a mayalso include one or more sensors 114 that are not included in the secondwearable device 102 c. Whether or not the first and second wearabledevice are identical, in some embodiments, the first wearable device 102a may be configured as a receiver and the second wearable device 102 cmay be configured as a transmitter (or vice versa).

It is noted that FIG. 6C may represent the user touching his leftforearm or FIG. 6C may represent the user hovering his finger above hisleft forearm. It is also noted that signals generated by the secondwearable device 102 c, at least in some instances, travel up the user'sright arm, across the user's body, and down the user's left arm to bereceived by the first wearable device 102 a. Thus, even if the user isnot touching his left forearm, the first wearable device 102 a is stillable to detect signals generated by the second wearable device 102 c.Importantly, by contacting his left forearm (or merely bringing hisfinger close to the left forearm), the user interferes with the signalstraveling across his or her body (e.g., the signals that travel up theright arm and eventually down the left arm). In some instances, thefirst wearable device 102 a is able to detect this interference anddetermine whether the detected interference satisfies contact criterion.Moreover, a magnitude of the detected interference may correspond to aparticular event. For example, a large magnitude difference (i.e., alarge interference) indicates that a touch occurred on the left forearmwhile a smaller magnitude difference (relative to the large magnitudedifferent) (i.e., a small interference) may indicate that a hover eventoccurred. It is noted that hover events can be detected in a varietyother ways as well. For example, in addition to the signals that travelup the user's right arm, across the user's body, and down the user'sleft arm to be received by the first wearable device 102 a, othersignals generated by the second wearable device 102 b can becomecapacitively coupled through the air when hand hovering occurs (e.g.,right hand hovers above left arm). The capacitive coupling is detectableand can be classified as a “hover.” The capacitive coupling reading canhave significant noise, which can contribute to the hover classification(e.g., noise is a factor considered when classifying an event as a hoverevent). Once the touch is made, the strength of the signal increases(e.g., significant jump detected) and can be classified as a “touch.”Additionally, the amount of noise decreases.

FIG. 6D shows one example signal pathway 640 established between twowearable devices. In this example, the second wearable device 102 c istransmitting one or more signals that couple into the wrist of the userand propagate (e.g., radiate) throughout the user's body. The firstwearable device 102 a receives at least some of the one or more signalstransmitted by the second wearable device 102 c, and in doing so,establishes the signal pathway 640 between the first wearable device 102a and the second wearable device 102 c. It is noted that in otherembodiments, the first wearable device 102 a is the transmitter and thesecond wearable device 102 c is the receiver. In such embodiments, thesignal pathway 640 would be reversed (e.g., signals travel from left toright). The signal pathway 640 is discussed in further detail below withreference to FIGS. 9 and 10.

FIG. 7 is a flow diagram illustrating a method of projecting images ontoa user's body in accordance with some embodiments. The steps of themethod 700 may be performed by a first wearable device (e.g., a wearabledevice 102 a, FIGS. 1A-1B), a second wearable device (e.g., wearabledevice 102 b, FIGS. 1A-1B), and a computer system (e.g., computer system130, FIGS. 1A-1B). FIG. 7 corresponds to instructions stored in acomputer memory or computer readable storage medium (e.g., memory 106 ofthe wearable device 102). For example, the operations of method 700 areperformed, at least in part, by a communication module (e.g.,communication module 218, FIG. 2), a projection generation module (e.g.,projection module 232, FIG. 2), gesture detection modules (e.g., tactilegesture detection module 234, computer vision gesture detection module238, FIG. 2), and/or location information modules (e.g., locationinformation 236, 240, FIG. 2).

At a first wearable device (e.g., wearable device 102 a) having aprojector (e.g., projector 115, FIG. 2) and a plurality of transducers(e.g., transducers 410, FIG. 4) the first wearable device projects 702(e.g., using projection module 232, FIG. 2), an image onto a portion ofa first appendage (e.g., forearm) of a user of the first wearabledevice. The method further includes detecting 704 (e.g., via tactilegesture detection module 234) a touch gesture on the image by a secondappendage of the user (e.g., a finger) distinct from the firstappendage.

The method further comprises at a second wearable device (e.g., wearabledevice 102 b) having a camera and a processor, determining 706 (e.g.,via computer vision gesture detection module 238, FIG. 2) a location(e.g., via location information 240, FIG. 2) of the touch gesture on theimage. In some embodiments, the second wearable device is an example thehead-mounted device 140, the computer system 130, or a combinationthereof. In some embodiments, the second wearable device (e.g., wearabledevice 102 b) is integrated with one or more of the head-mounted device140 and the computer system 130. In some other embodiments, the secondwearable device (e.g., wearable device 102 b) is distinct from thehead-mounted device 140 and the computer system 130. In one example, thesecond wearable device is the AR system 1100 or the VR system 1200.

In some embodiments, the second wearable device confirms 708 that thedetected touch gesture has occurred on the image by the second appendageof the user.

The method further comprises a computer system (e.g., computer system130, FIGS. 1A-1B) is instructed to perform 710 an operation inaccordance with the detecting and the location. In some embodiments, thecomputer system performs 712 the operation in accordance with theconfirming that the detected touch gesture has occurred on the image bythe second appendage of the user. For example, with reference to FIG.6A, the first wearable device 102 a projects the image 602 (e.g., userinterface) onto the user's left forearm, and a second wearable device102 b worn on the user's head captures, via the camera 118, the user'sright index finger interacting with the projected image 602. In thisway, the second wearable device 102 b determines a location of the touchgesture on the image (e.g., touch location 604 in FIG. 6B).Additionally, the first wearable device 102 a is able to detect (sense)the right index finger interacting with the projected image 602 (e.g.,sense the touch on the left forearm). Thus, the first and secondwearable devices work together to detect the touch gesture on the image.Furthermore, the location of the touch gesture on the image maycorrespond to an affordance (or some other interface input), and theoperation is associated with the affordance.

FIG. 8 is a flow diagram illustrating a method of projecting images ontoa user's body in accordance with some embodiments. The steps of themethod 800 may be performed by a first wearable device (e.g., a wearabledevice 102 a, FIGS. 1A-1B), a second wearable device (e.g., wearabledevice 102 b, FIGS. 1A-1B), a third wearable device (e.g., wearabledevice 102 c, FIG. 1B) and a computer system (e.g., computer system 130,FIGS. 1A-1B). FIG. 8 corresponds to instructions stored in a computermemory or computer readable storage medium (e.g., memory 106 of thewearable device 102). For example, the operations of method 800 areperformed, at least in part, by a communication module (e.g.,communication module 218, FIG. 2), a projection generation module (e.g.,projection module 232, FIG. 2), gesture detection modules (e.g., tactilegesture detection module 234, computer vision gesture detection module238, FIG. 2), and/or location information modules (e.g., locationinformation 236, 240, FIG. 2). It is noted that the steps of the method800 can be performed in conjunction with the steps in method 700.

In some embodiments, a first wearable device (e.g., wearable device 102a) projects 802 an image onto a portion of a first appendage of a userof the first wearable device. In some embodiments, the first wearabledevice generates 804 signals that couple/vibrate into at least a portionof the first appendage of the user of the first wearable device. Forexample, FIG. 6C shows a first wearable device on the left wrist of theuser which may generate signals that vibrate through the user's leftarm/wrist/hand/fingers.

In some embodiments, a third wearable device (e.g., wearable device 102c) receives 812 at least a portion of the signals generated by the firstplurality of transducers when the first appendage is within a thresholddistance from the third wearable device. For example, FIG. 6C shows auser having a first wearable device on the left wrist, and a secondwearable device on the right wrist. The first wearable device generatessignals through the left wrist which the second wearable device on theright wrist receives when the first and second wearable devices areproximate to each other.

In some embodiments, the third wearable device determines 814 a positionof a portion of the first appendage with respect to a portion of thethird wearable device. For example, the third wearable device may havetransducers that receives signals from the first wearable device atspecific locations of the third wearable device. The received signalinformation may be analyzed by the control circuit of the wearabledevice to determine a position of a portion of the left arm with respectto the right arm.

In some embodiments, a second wearable device determines 808 a locationof the touch gesture on the image. In some embodiments, the secondwearable device confirms 810 that the detected touch gesture hasoccurred on the image by the second appendage of the user.

In some embodiments, a computer system (e.g., computer system 130, FIGS.1A-1B) is instructed to perform 816 an operation in accordance with thedetecting, the position, and the location. In some embodiments, thecomputer system performs 818 the operation in accordance with theconfirming that the detected touch gesture has occurred on the image bythe second appendage of the user.

FIG. 9 is a high level flow diagram illustrating a method 900 ofdetecting a touch on a user's body in accordance with some embodiments.The steps of the method 900 may be performed by a first wearable device(e.g., instance of wearable device 102), a second wearable device (e.g.,instance of wearable device 102), and a computer system (e.g., computersystem 130). FIG. 9 corresponds to instructions stored in a computermemory or computer readable storage medium. For ease of discussion, thefirst wearable device is attached to a first appendage of the user, suchas the user's wrist. In some embodiments, the second wearable device isalso attached to the first appendage, while in other embodiments thesecond wearable device is attached elsewhere on the user's body (e.g.,the user's other wrist, on the user's head, or various other places onthe user's body). It is also noted that, in some embodiments, thecomputer system and the second wearable device may be part of the samedevice, while in other embodiments the computer system and the secondwearable device are separate from each other.

In some embodiments, the method 900 begins with the computer systeminitiating (902) a signal transmission from the second wearable device.For example, the computer system may provide instructions to the secondwearable device to emit one or more signals (e.g., acoustic waves). Insome embodiments, the computer system initiates the signal transmissionwhen the computer system is powered up. The computer system may alsoinitiate the signal transmission when a trigger event occurs in a VR/ARapplication being run by the computer system. For example, when a userreaches a particular stage in a video game, the computer system mayinitiate the signal transmission.

The method 900 may also include the computer system providinginstructions to a head-mounted display (e.g., head-mounted display 140,FIG. 1A) to display a user interface or other graphics/image(s) (e.g.,interface 602, FIG. 6A) on the first appendage of the user. As discussedabove, the user interface may be projected onto the first appendage orpresented, using augmented reality, via the head-mounted display so thatthe user perceives the user interface on the first appendage. In someembodiments, the user may perform an action that triggers display of theuser interface (and initiation of the signal transmission). For example,with reference to FIG. 6A, the user may move his arm and head to displaypositions (e.g., eyes pointed/aimed towards forearm while forearm ispositioned in a viewing position).

Additionally, the computer system may receive motion information fromsensors 114 of the first wearable device indicating that the user hasmoved (e.g., rotated) the first appendage while, in augmented reality(or virtual reality), a user interface (or some other augmented object)is being displayed on the user's first appendage. In such instances, thecomputer system generates content for the head-mounted display thatmirrors and/or otherwise accounts for the user's movement in theaugmented environment (e.g., the user interface in FIG. 6A rotates inaccordance with the rotation of the user's forearm). In this way, thedisplayed user interface appears fixed to the first appendage. Moreover,the computer system may receive motion information from sensors 145 ofthe head-mounted display indicating that the user has moved his headwhile, in augmented reality (or virtual reality), the user interface isbeing displayed on the user's first appendage. In such instances, thecomputer system generates content for the head-mounted display thatmirrors the user's movement in the augmented environment.

In some embodiments, the first and second wearable devices aretransmitters and receivers. For example, when the first wearable deviceis attached to the user's left wrist and the second wearable device isattached to the user's right wrist, the first wearable device may act asa receiver and the second wearable device may act as a transmitter infirst circumstances (e.g., when the user has his left arm and head infirst display positions), while the first wearable device may act as atransmitter and the second wearable device may act as a receiver insecond circumstances (e.g., when the user has his right arm and head insecond display positions). In this way, the user can intuitively displaya user interface (e.g., interface 602) on his left arm when desired, andleverage the first wearable device to confirm touches on the left arm,and also display another interface (or the same interface) on his rightarm when desired, and leverage the second wearable device to confirmtouches on the right arm. Accordingly, in some embodiments, the computersystem determines that a respective appendage is in a predetermineddisplay position (e.g., using at least the cameras 139) and that theuser's head is aimed towards the respective appendage (e.g., using atleast the cameras 139 or other sensors), and in response to making thesedeterminations, the computer system instructs the head-mounted displayto display a user interface on the respective appendage (e.g., as shownin FIG. 6A) and also instructs (step 902) at least one wearable deviceto emit signals, as described below.

The method 900 includes the second wearable device emitting (904) theone or more signals that propagate through at least the first appendageof the user (e.g., in response to receiving the instructions from thecomputer system). The method 900 also includes the first wearable devicereceiving (906) at least some of the one or more signals emitted by thesecond wearable device. In some embodiments, reception of the signals bythe first wearable device establishes (908) a signal pathway between thefirst wearable device and the second wearable device. Furthermore, afterestablishing the signal path and while the second wearable devicecontinues to emit the one or more signals, the method 900 furtherincludes the first wearable device sensing (910) a change in the signalpathway (e.g., values of the set of signals received by the one or moretransducers change). The sensed change in the signal pathway may beattributed to a touch event on the user's first appendage (e.g., usertouches left forearm with right index finger) that interferes with theestablished signal pathway. In such a case, the first wearable devicereports (912) a candidate touch event to the computer system.Alternatively, the sensed change in the signal pathway may be attributedto noise or some other non-touch event. In such a case, the firstwearable device forgoes reporting a candidate touch event and continuesto monitor the signal pathway. Method 1000, discussed below, describesthe operations of method 900 performed by the first wearable device inmore detail.

The method 900 further includes the computer system capturing (914) thecandidate touch event. For example, one or more of the cameras 139 ofthe computer system 130 may capture the user's right index fingermovement towards the user's left forearm, as illustrated in FIG. 6A. Insome instances, capturing the touch candidate event includes capturing alocation of the touch with respect to the displayed user interface.

In response to capturing the candidate touch event and receiving (916)the report of the candidate touch event from the first wearable device,the method 900 includes the computer system executing (918) a functionassociated with the candidate touch event. The computer system executesthe function if it determines that the capturing of the candidate touchevent and the report of the candidate touch event align (e.g., align intime and space). In other words, the report of the candidate touch eventfrom the first wearable device is used by the computer system to confirmwhat the one or more cameras 139 captured (e.g., the cameras 139 maycapture the touch location 604 (FIG. 6C) and the report of the candidatetouch event from the first wearable device confirms contact with theleft appendage). In this way, the computer system is able to distinguisha finger hovering above the user interface and a finger attempting tointeract with the user interface (e.g., when a user merely hovers hisfinger above the interface, the cameras 139 alone struggle todistinguish the hovering from an actual touch event). In someembodiments, the displayed user interface includes one or moreaffordances, and the capturing of the candidate touch event indicatesthat the user intended to interact with a first affordance of the one ormore affordances (e.g., interface 602 in FIG. 6A includes multipleaffordances). In such embodiments, executing the function associatedwith the candidate touch event includes executing a function associatedwith the first affordance.

FIG. 10 is a flow diagram illustrating a method 1000 of confirming atouch on a user's body in accordance with some embodiments. The steps ofthe method 1000 may be performed by a first wearable device (e.g.,instance of wearable device 102) that includes one or more transducers(e.g., transducers 410, FIG. 4) (1001). FIG. 10 corresponds toinstructions stored in a computer memory or computer readable storagemedium. For ease of discussion, the first wearable device is attached toa first appendage of the user, such as the user's wrist. It is notedthat the steps of the method 1000 can be performed in conjunction withthe steps in methods 700, 800, and 900.

In some embodiments, the method 1000 includes receiving (1002), by theone or more transducers of the first wearable device, a set of signalstransmitted by a second wearable device attached to the user, where (i)receiving the set of signals creates a signal pathway between the firstand second wearable devices, and (ii) signals in the set of signalspropagate through at least the user's first appendage. In someembodiments, the second wearable device is also attached to the firstappendage, while in other embodiments the second wearable device isattached elsewhere on the user's body (e.g., on another appendage or theuser's head). To illustrate the signal pathway, with reference to FIG.6C, signals generated by the second wearable device 102 c, at least insome instances, travel (e.g., propagate, radiate) up the user's rightarm, across the user's body, and down the user's left arm to be receivedby the first wearable device 102 a (the generated signals at radiatetowards the user's right-hand fingers). Thus, even if the user is nottouching his left forearm with his right hand, the first wearable device102 a is still able to detect signals generated by the second wearabledevice 102 c. It is noted that the signals generated by the secondwearable device 102 c may propagate (radiate) throughout the user'sentire body.

In some embodiments, the method 1000 includes determining (1004)baseline characteristics for the signal pathway created between thefirst wearable device and the second wearable device. The baselinecharacteristics may include values for phase, amplitude, frequency, etc.associated with the signals received by the first wearable device.Accordingly, in some embodiments, the baseline characteristics include abaseline phase value (1006), a baseline amplitude phase (1008), and/or abaseline frequency value (among other waveform characteristics). In someembodiments, the baseline characteristics are determined during acalibration process of the user, and it is noted that baselinecharacteristics may vary from user to user based on bodily differencesbetween users (e.g., bodily tissue and bone structure will vary fromuser to user, creating different impedances to the signals radiatingthrough the body).

In some embodiments, the method 1000 includes sensing (detecting,measuring) (1010) a change in the baseline characteristics whilereceiving the set of signals. For example, when the baselinecharacteristics include the baseline phase value, sensing the change inthe baseline characteristics for the signal pathway includes detecting aphase value of the signal pathway that differs from the baseline phasevalue. In another example (or in addition to the previous example), whenthe baseline characteristics include the baseline amplitude value,sensing the change in the baseline characteristics for the signalpathway includes detecting an amplitude value of the signal pathway thatdiffers from the baseline amplitude value.

In some embodiments, the method 1000 includes determining (1012) whetherthe sensed change in the baseline characteristics satisfies a contactcriterion (or in some embodiments, contact criteria). To provide somecontext, in some instances, the sensed change in the baselinecharacteristics is attributable to a touch on the first appendage by theuser that interferes with the established signal pathway (e.g., thetouch impedes the signal pathway). Additionally, using the wearabledevice arrangement in FIG. 6C as an example, it is noted that when theuser touches his left forearm with his right index finger, additionalsignals may propagate from the right index finger into the left forearmat the touch location, thereby causing (or at minimum contributing to)the interference of the signal pathway. However, in some otherinstances, the sensed change in the baseline characteristics isattributable to noise, or the user hovering his finger above the firstappendage. Accordingly, the first wearable device compares the sensedchange with contact criterion to distinguish intended touches from otherevents.

In some embodiments, in accordance with a determination that the sensedchange in the baseline characteristics for the signal pathway does notsatisfy the contact criterion (1012—No), the method 1000 includescontinuing to sense for changes in the baseline characteristics that maysatisfy the contact criterion. Sensed changes in the baselinecharacteristics that do not satisfy the contact criterion may beattributable to noise or quick contacts (e.g., brushes) betweenappendages. In some embodiments, the contact criterion includes: (i) atouch criterion that is set to be satisfied by touches but not hoveringevents, and (ii) a hovering criterion that is set to be satisfied byhovering events but not touches. Various examples of the contactcriterion are provided below.

In some embodiments, in accordance with a determination that the sensedchange in the baseline characteristics for the signal pathway satisfiesthe contact criterion (1012—Yes), the method 1000 includes reporting(1016) a candidate touch event on the user's first appendage. In someembodiments, reporting the candidate touch event includes sending amessage (e.g., a report) to a computer system (e.g., computer system130) that a candidate touch event was sensed and confirmed (i.e., atouch confirmed flag is sent). The message may also include a timestampof when the change in the baseline characteristics were sensed (and/or aduration of the sensed change). The candidate touch event may be any oneof a tap gesture, a swipe gesture, a pinch gesture, a pull gesture, or atwist gesture. It is also noted that, in some embodiments, the computersystem and the second wearable device may be part of the same device,while in other embodiments the computer system and the second wearabledevice are separate from each other.

In some embodiments, the contact criterion includes a phase differencethreshold. In such embodiments, reporting the candidate touch event isperformed in accordance with a determination that a difference betweenthe detected phase value (from step 1010) and the baseline phase valuesatisfies the phase difference threshold. In some embodiments, thecontact criterion includes an amplitude difference threshold. In suchembodiments, reporting the candidate touch event is performed inaccordance with a determination that a difference between the detectedamplitude value (from step 1010) and the baseline amplitude valuesatisfies the amplitude difference threshold. In another embodiment, thecontact criteria include an amplitude difference threshold and a phasedifference threshold. In such embodiments, reporting the candidate touchevent is performed in accordance with a determination that: (i) adifference between the detected amplitude value and the baselineamplitude value satisfies the amplitude difference threshold, and (ii) adifference between the detected phase value and the baseline phase valuesatisfies the phase difference threshold. In addition to or separatelyfrom the embodiments above, the contact criterion may also include atime threshold. In such embodiments, sensing (1010) the change in thebaseline characteristics includes sensing the change for a period oftime (i.e., a duration of the sensed change is determined) and reportingthe candidate touch event is performed in accordance with adetermination that the period of time satisfies the time threshold. Thetime threshold can be used to filter out noise, as well asaccidental/unintentional touching of the first appendage (e.g., a quickbrushing of the first appendage with the second appendage can befiltered out).

As noted above, the contact criterion may include a touch criterion anda hovering criterion. In some embodiments, the contact criterionincludes a first phase difference threshold for the touch criterion anda second phase difference threshold for the hovering criterion. Thefirst phase difference threshold differs (e.g., is larger than) fromthat of the second phase difference threshold. Accordingly, in someembodiments, the method 1000 includes reporting a candidate hoveringevent in accordance with a determination that a difference between thedetected phase value (from step 1010) and the baseline phase valuesatisfies the second phase difference threshold but does not satisfy thefirst phase threshold. The contact criterion can also include a firstamplitude difference threshold for the touch criterion and a secondamplitude difference threshold for the hovering criterion. The firstamplitude difference threshold differs (e.g., is larger than) from thatof the second amplitude difference threshold. The steps below discusshow the “candidate touch event” can similarly be performed incircumstances when the first wearable device reports a candidatehovering event instead of the candidate touch event.

In some embodiments, reporting the candidate touch event includessending transducer (and/or sensor) data corresponding to the sensedchange in the baseline characteristics to the computer system (e.g.,computer system 130) (1018). In some embodiments, the transducer (and/orsensor) data includes a time stamp (and/or a duration) associated withsensing the change in the baseline characteristics. Additionally, insome embodiments, the transducer (and/or sensor) data includes thesensed change in the baseline characteristics (e.g., values for phase,amplitude, etc.). In some embodiments, the computer system uses thetransducer (and/or sensor) data to confirm that a touch event occurred.Furthermore, in some embodiments, the computer system uses thetransducer (and/or sensor) data to determine an approximate location ofthe candidate touch event on the user's first appendage. For example,the values for phase, amplitude, etc. may indicate the approximatelocation of the candidate touch event on the user's first appendage.

As discussed above with reference to the method 900, the computer systemmay display, on the user's first appendage, a user interface thatincludes one or more affordances, and the candidate touch event may beassociated with a first affordance of the one or more affordancesincluded in the user interface. The computer system may use thedata/information/report associated with the sensed change in thebaseline characteristics to confirm whether a touch event with thedisplayed user interface occurred. Put another way, the computer systemdetermines whether the user intended to interact with an affordance ofthe user interface displayed on the user's first appendage based, atleast in part, on the data/information/report associated with the sensedchange in the signal pathway received from the first wearable device.

In addition, the computer system may capture, via one or more cameras(e.g., cameras 139, FIG. 1A), the candidate touch event, and generateimage data according to the capturing of the candidate touch event. Inthis way, the computer system may determine an approximate location ofthe candidate touch event on the user's first appendage based on theimage data. Additionally, the computer system may determine whether thecaptured movements of the user amount to a candidate touch event (e.g.,the second appendage comes within a threshold distance from the firstappendage, or the second appendage visually obstructs a portion of thefirst appendage where the user interface is displayed). The computersystem may then use data/information/report received from the firstwearable device (step 1012—Yes) to confirm that the candidate touchevent is an actual touch event. The computer system then executes afunction associated with the first affordance of the user interface (i)if the data/information/report associated with the sensed change in thesignal pathway received from the first wearable device confirms that theuser intended to interact with the first affordance and (ii) if theimage data confirms that the user intended to interact with the firstaffordance. This dual confirmation process creates a robust approach todetecting touch events (and hovering events) with artificial userinterfaces.

In some embodiments, if the computer system receives the reporting ofthe candidate touch event from the first wearable device within a timewindow of capturing the candidate touch event, the computer systemexecutes the function. The time window may be a predefined time window.In some embodiments, the computer system tracks a time frame of when thecandidate touch event could have occurred, based on the image data(i.e., generates a time frame of when the touch event could haveoccurred). In such embodiments, if the time stamp included in thedata/information/report received from the first wearable device fallswithin the time frame, the computer system executes the function. It isnoted that each of the one or more affordances of the displayed userinterface may have a unique function.

In some embodiments, the transducer data sent to the computer systemfurther includes information indicating an approximate location of thecandidate touch event (1020). For example, the first wearable device maydetermine (1014) the approximate location of the candidate touch eventon the user's first appendage based, at least in part, on the sensedchange in the baseline characteristics. This can be accomplished byevaluating a phase value (and/or an amplitude value) of the signalpathway. For example, one or more first phase values (and/or one or morefirst amplitude values) may indicate that the user touched close to hiswrist, whereas one or more second phase values (and/or one or moresecond amplitude values) different from the one or more first phasevalues (or the one or more first amplitude values) may indicate that theuser touched close to his elbow. Additionally, in some embodiments, thefirst wearable device may include one or more cameras 118 that capturethe candidate touch event.

In some embodiments, instead of the first wearable device monitoringchanges in an established signal path, the first wearable devicedetermines that a candidate touch event has occurred based on thebaseline characteristics of the signal pathway. In other words, valuesof phase, amplitude, etc. of the set of signals received by the one ormore transducers themselves may indicate that a candidate touch eventhas occurred.

Embodiments of the instant disclosure may include or be implemented inconjunction with various types of artificial reality systems. Artificialreality may constitute a form of reality that has been altered byvirtual objects for presentation to a user. Such artificial reality mayinclude and/or represent VR, AR, MR, hybrid reality, or some combinationand/or variation of one or more of the same. Artificial reality contentmay include completely generated content or generated content combinedwith captured (e.g., real-world) content. The artificial reality contentmay include video, audio, haptic feedback, or some combination thereof,any of which may be presented in a single channel or in multiplechannels (such as stereo video that produces a three-dimensional effectto a viewer). Additionally, in some embodiments, artificial reality mayalso be associated with applications, products, accessories, services,or some combination thereof, that are used to, e.g., create content inan artificial reality and/or are otherwise used in (e.g., to performactivities in) an artificial reality.

Artificial reality systems may be implemented in a variety of differentform factors and configurations. Some artificial reality systems may bedesigned to work without near-eye displays (NEDs), an example of whichis AR system 1100 in FIG. 11. Other artificial reality systems mayinclude an NED that also provides visibility into the real world (e.g.,AR system 1200 in FIG. 12) or that visually immerses a user in anartificial reality (e.g., VR system 1300 in FIG. 13). While someartificial reality devices may be self-contained systems, otherartificial reality devices may communicate and/or coordinate withexternal devices to provide an artificial reality experience to a user.Examples of such external devices include handheld controllers, mobiledevices, desktop computers, devices worn by a user (e.g., wearabledevice 102 a, wearable device 102 b, . . . wearable device 102 n),devices worn by one or more other users, and/or any other suitableexternal system.

FIGS. 11-13 provide additional examples of the devices used in thesystem 100. AR system 1100 in FIG. 11 generally represents a wearabledevice dimensioned to fit about a body part (e.g., ahead) of a user. TheAR system 1100 may include the functionality of the wearable device 102,and may include additional functions. As shown, the AR system 1100includes a frame 1102 (e.g., band) and a camera assembly 1104 that iscoupled to frame 1102 and configured to gather information about a localenvironment by observing the local environment. The AR system 1100 mayalso include one or more transducers (e.g., instances of the transducers410, FIG. 4). In one example, the AR system 1100 includes outputtransducers 1108(A) and 1108(B) and input transducers 1110. Outputtransducers 1108(A) and 1108(B) may provide audio feedback, hapticfeedback, and/or content to a user, and input audio transducers maycapture audio (or other signals/waves) in a user's environment. In someembodiments, the camera assembly 1104 includes one or more projectors(e.g., projectors 115) that allows the AR system 1100 to project images(e.g., if the AR system 1100 is worn on the user's wrist, then thecamera assembly 1104 can project images onto the user's wrist andforearm).

Thus, the AR system 1100 does not include a near-eye display (NED)positioned in front of a user's eyes. AR systems without NEDs may take avariety of forms, such as head bands, hats, hair bands, belts, watches,wrist bands, ankle bands, rings, neckbands, necklaces, chest bands,eyewear frames, and/or any other suitable type or form of apparatus.While the AR system 1100 may not include an NED, the AR system 1100 mayinclude other types of screens or visual feedback devices (e.g., adisplay screen integrated into a side of frame 1102).

The embodiments discussed in this disclosure may also be implemented inAR systems that include one or more NEDs. For example, as shown in FIG.12, the AR system 1200 may include an eyewear device 1202 with a frame1210 configured to hold a left display device 1215(A) and a rightdisplay device 1215(B) in front of a user's eyes. Display devices1215(A) and 1215(B) may act together or independently to present animage or series of images to a user. While the AR system 1200 includestwo displays, embodiments of this disclosure may be implemented in ARsystems with a single NED or more than two NEDs.

In some embodiments, the AR system 1200 may include one or more sensors,such as sensor 1240. Sensor 1240 may generate measurement signals inresponse to motion of AR system 1200 and may be located on substantiallyany portion of frame 1210. Sensor 1240 may include a position sensor, aninertial measurement unit (IMU), a depth camera assembly, or anycombination thereof. In some embodiments, the AR system 1200 may or maynot include sensor 1240 or may include more than one sensor. Inembodiments in which sensor 1240 includes an IMU, the IMU may generatecalibration data based on measurement signals from sensor 1240. Examplesof sensor 1240 may include, without limitation, accelerometers,gyroscopes, magnetometers, other suitable types of sensors that detectmotion, sensors used for error correction of the IMU, or somecombination thereof. Sensors are also discussed above with reference toFIG. 1 (e.g., sensors 145 of the head-mounted display 140).

The AR system 1200 may also include a microphone array with a pluralityof acoustic sensors 1220(A)-1220(J), referred to collectively asacoustic sensors 1220. Acoustic sensors 1220 may be transducers thatdetect air pressure variations induced by sound waves. Each acousticsensor 1220 may be configured to detect sound and convert the detectedsound into an electronic format (e.g., an analog or digital format). Themicrophone array in FIG. 12 may include, for example, ten acousticsensors: 1220(A) and 1220(B), which may be designed to be placed insidea corresponding ear of the user, acoustic sensors 1220(C), 1220(D),1220(E), 1220(F), 1220(G), and 1220(H), which may be positioned atvarious locations on frame 1210, and/or acoustic sensors 1220(I) and1220(J), which may be positioned on a corresponding neckband 1205. Insome embodiments, the neckband 1205 is an example of the computer system130.

The configuration of acoustic sensors 1220 of the microphone array mayvary. While the AR system 1200 is shown in FIG. 12 as having tenacoustic sensors 1220, the number of acoustic sensors 1220 may begreater or less than ten. In some embodiments, using higher numbers ofacoustic sensors 1220 may increase the amount of audio informationcollected and/or the sensitivity and accuracy of the audio information.In contrast, using a lower number of acoustic sensors 1220 may decreasethe computing power required by a controller 1250 to process thecollected audio information. In addition, the position of each acousticsensor 1220 of the microphone array may vary. For example, the positionof an acoustic sensor 1220 may include a defined position on the user, adefined coordinate on the frame 1210, an orientation associated witheach acoustic sensor, or some combination thereof.

Acoustic sensors 1220(A) and 1220(B) may be positioned on differentparts of the user's ear, such as behind the pinna or within the auricleor fossa. Or, there may be additional acoustic sensors on or surroundingthe ear in addition to acoustic sensors 1220 inside the ear canal.Having an acoustic sensor positioned next to an ear canal of a user mayenable the microphone array to collect information on how sounds arriveat the ear canal. By positioning at least two of acoustic sensors 1220on either side of a user's head (e.g., as binaural microphones), the ARdevice 1200 may simulate binaural hearing and capture a 3D stereo soundfield around about a user's head. In some embodiments, the acousticsensors 1220(A) and 1220(B) may be connected to the AR system 1200 via awired connection, and in other embodiments, the acoustic sensors 1220(A)and 1220(B) may be connected to the AR system 1200 via a wirelessconnection (e.g., a Bluetooth connection). In still other embodiments,acoustic sensors 1220(A) and 1220(B) may not be used at all inconjunction with the AR system 1200.

Acoustic sensors 1220 on frame 1210 may be positioned along the lengthof the temples, across the bridge, above or below display devices1215(A) and 1215(B), or some combination thereof. Acoustic sensors 1220may be oriented such that the microphone array is able to detect soundsin a wide range of directions surrounding the user wearing AR system1200. In some embodiments, an optimization process may be performedduring manufacturing of AR system 1200 to determine relative positioningof each acoustic sensor 1220 in the microphone array.

The AR system 1200 may further include or be connected to an externaldevice (e.g., a paired device), such as neckband 1205. As shown,neckband 1205 may be coupled to eyewear device 1202 via one or moreconnectors 1230. Connectors 1230 may be wired or wireless connectors andmay include electrical and/or non-electrical (e.g., structural)components. In some cases, eyewear device 1202 and neckband 1205 mayoperate independently without any wired or wireless connection betweenthem. While FIG. 12 illustrates the components of eyewear device 1202and neckband 1205 in example locations on eyewear device 1202 andneckband 1205, the components may be located elsewhere and/ordistributed differently on eyewear device 1202 and/or neckband 1205. Insome embodiments, the components of eyewear device 1202 and neckband1205 may be located on one or more additional peripheral devices pairedwith eyewear device 1202, neckband 1205, or some combination thereof.Furthermore, neckband 1205 generally represents any type or form ofpaired device. Thus, the following discussion of neckband 1205 may alsoapply to various other paired devices, such as smart watches, smartphones, wrist bands, other wearable devices, hand-held controllers,tablet computers, laptop computers, etc.

Pairing external devices, such as neckband 1205, with AR eyewear devicesmay enable the eyewear devices to achieve the form factor of a pair ofglasses while still providing sufficient battery and computation powerfor expanded capabilities. Some or all of the battery power,computational resources, and/or additional features of the AR system1200 may be provided by a paired device or shared between a paireddevice and an eyewear device, thus reducing the weight, heat profile,and form factor of the eyewear device overall while still retainingdesired functionality. For example, neckband 1205 may allow componentsthat would otherwise be included on an eyewear device to be included inneckband 1205 since users may tolerate a heavier weight load on theirshoulders than they would tolerate on their heads. Neckband 1205 mayalso have a larger surface area over which to diffuse and disperse heatto the ambient environment. Thus, neckband 1205 may allow for greaterbattery and computation capacity than might otherwise have been possibleon a stand-alone eyewear device. Since weight carried in neckband 1205may be less invasive to a user than weight carried in eyewear device1202, a user may tolerate wearing a lighter eyewear device and carryingor wearing the paired device for greater lengths of time than the userwould tolerate wearing a heavy standalone eyewear device, therebyenabling an artificial reality environment to be incorporated more fullyinto a user's day-to-day activities.

Neckband 1205 may be communicatively coupled with eyewear device 1202and/or to other devices. The other devices may provide certain functions(e.g., tracking, localizing, depth mapping, processing, storage, etc.)to the AR system 1200. In the embodiment of FIG. 12, neckband 1205 mayinclude two acoustic sensors (e.g., 1220(I) and 1220(J)) that are partof the microphone array (or potentially form their own microphonesubarray). Neckband 1205 may also include a controller 1225 and a powersource 1235.

Acoustic sensors 1220(I) and 1220(J) of neckband 1205 may be configuredto detect sound and convert the detected sound into an electronic format(analog or digital). In the embodiment of FIG. 12, acoustic sensors1220(I) and 1220(J) may be positioned on neckband 1205, therebyincreasing the distance between neckband acoustic sensors 1220(I) and1220(J) and other acoustic sensors 1220 positioned on eyewear device1202. In some cases, increasing the distance between acoustic sensors1220 of the microphone array may improve the accuracy of beamformingperformed via the microphone array. For example, if a sound is detectedby acoustic sensors 1220(C) and 1220(D) and the distance betweenacoustic sensors 1220(C) and 1220(D) is greater than, e.g., the distancebetween acoustic sensors 1220(D) and 1220(E), the determined sourcelocation of the detected sound may be more accurate than if the soundhad been detected by acoustic sensors 1220(D) and 1220(E).

Controller 1225 of neckband 1205 may process information generated bythe sensors on neckband 1205 and/or AR system 1200. For example,controller 1225 may process information from the microphone array thatdescribes sounds detected by the microphone array. For each detectedsound, controller 1225 may perform a direction of arrival (DOA)estimation to estimate a direction from which the detected sound arrivedat the microphone array. As the microphone array detects sounds,controller 1225 may populate an audio data set with the information. Inembodiments in which AR system 1200 includes an IMU, controller 1225 maycompute all inertial and spatial calculations from the IMU located oneyewear device 1202. Connector 1230 may convey information between ARsystem 1200 and neckband 1205 and between AR system 1200 and controller1225. The information may be in the form of optical data, electricaldata, wireless data, or any other transmittable data form. Moving theprocessing of information generated by AR system 1200 to neckband 1205may reduce weight and heat in eyewear device 1202, making it morecomfortable to a user.

Power source 1235 in neckband 1205 may provide power to eyewear device1202 and/or to neckband 1205. Power source 1235 may include, withoutlimitation, lithium-ion batteries, lithium-polymer batteries, primarylithium batteries, alkaline batteries, or any other form of powerstorage. In some cases, power source 1235 may be a wired power source.Including power source 1235 on neckband 1205 instead of on eyeweardevice 1202 may help better distribute the weight and heat generated bypower source 1235.

As noted, some artificial reality systems may, instead of blending anartificial reality with actual reality, substantially replace one ormore of a user's sensory perceptions of the real world with a virtualexperience. One example of this type of system is a head-worn displaysystem, such as VR system 1300 in FIG. 13, that mostly or completelycovers a user's field of view. VR system 1300 may include a front rigidbody 1302 and a band 1304 shaped to fit around a user's head. VR system1300 may also include output audio transducers 1306(A) and 1306(B).Furthermore, while not shown in FIG. 13, front rigid body 1302 mayinclude one or more electronic elements, including one or moreelectronic displays, one or more IMUs, one or more tracking emitters ordetectors, and/or any other suitable device or system for creating anartificial reality experience. Although not shown, the VR system 1300may include the computer system 130.

Artificial reality systems may include a variety of types of visualfeedback mechanisms. For example, display devices in AR system 1200and/or VR system 1300 may include one or more liquid-crystal displays(LCDs), light emitting diode (LED) displays, organic LED (OLED)displays, and/or any other suitable type of display screen. Artificialreality systems may include a single display screen for both eyes or mayprovide a display screen for each eye, which may allow for additionalflexibility for varifocal adjustments or for correcting a user'srefractive error. Some artificial reality systems may also includeoptical subsystems having one or more lenses (e.g., conventional concaveor convex lenses, Fresnel lenses, adjustable liquid lenses, etc.)through which a user may view a display screen.

In addition to or instead of using display screens, some artificialreality systems may include one or more projection systems. For example,display devices in AR system 1200 and/or VR system 1300 may includemicro-LED projectors that project light (using, e.g., a waveguide) intodisplay devices, such as clear combiner lenses that allow ambient lightto pass through. The display devices may refract the projected lighttoward a user's pupil and may enable a user to simultaneously view bothartificial reality content and the real world. Artificial realitysystems may also be configured with any other suitable type or form ofimage projection system.

Artificial reality systems may also include various types of computervision components and subsystems. For example, AR system 1100, AR system1200, and/or VR system 1300 may include one or more optical sensors suchas two-dimensional (2D) or three-dimensional (3D) cameras,time-of-flight depth sensors, single-beam or sweeping laserrangefinders, 3D LiDAR sensors, and/or any other suitable type or formof optical sensor. An artificial reality system may process data fromone or more of these sensors to identify a location of a user, to mapthe real world, to provide a user with context about real-worldsurroundings, and/or to perform a variety of other functions.

Artificial reality systems may also include one or more input and/oroutput audio transducers. In the examples shown in FIGS. 11 and 13,output audio transducers 1108(A), 1108(B), 1106(A), and 1306(B) mayinclude voice coil speakers, ribbon speakers, electrostatic speakers,piezoelectric speakers, bone conduction transducers, cartilageconduction transducers, and/or any other suitable type or form of audiotransducer. Similarly, input audio transducers 1110 may includecondenser microphones, dynamic microphones, ribbon microphones, and/orany other type or form of input transducer. In some embodiments, asingle transducer may be used for both audio input and audio output.

The artificial reality systems shown in FIGS. 11-13 may include tactile(i.e., haptic) feedback systems, which may be incorporated intoheadwear, gloves, body suits, handheld controllers, environmentaldevices (e.g., chairs, floormats, etc.), and/or any other type of deviceor system, such as the wearable devices 102 discussed herein.Additionally, in some embodiments, the haptic feedback systems may beincorporated with the artificial reality systems (e.g., the AR system1100 may include the wearable device 102 (FIG. 1). Haptic feedbacksystems may provide various types of cutaneous feedback, includingvibration, force, traction, texture, and/or temperature. Haptic feedbacksystems may also provide various types of kinesthetic feedback, such asmotion and compliance. Haptic feedback may be implemented using motors,piezoelectric actuators, fluidic systems, and/or a variety of othertypes of feedback mechanisms. Haptic feedback systems may be implementedindependent of other artificial reality devices, within other artificialreality devices, and/or in conjunction with other artificial realitydevices.

By providing haptic sensations, audible content, and/or visual content,artificial reality systems may create an entire virtual experience orenhance a user's real-world experience in a variety of contexts andenvironments. For instance, artificial reality systems may assist orextend a user's perception, memory, or cognition within a particularenvironment. Some systems may enhance a user's interactions with otherpeople in the real world or may enable more immersive interactions withother people in a virtual world. Artificial reality systems may also beused for educational purposes (e.g., for teaching or training inschools, hospitals, government organizations, military organizations,business enterprises, etc.), entertainment purposes (e.g., for playingvideo games, listening to music, watching video content, etc.), and/orfor accessibility purposes (e.g., as hearing aids, vision aids, etc.).The embodiments disclosed herein may enable or enhance a user'sartificial reality experience in one or more of these contexts andenvironments and/or in other contexts and environments.

Some AR systems may map a user's environment using techniques referredto as “simultaneous location and mapping” (SLAM). SLAM mapping andlocation identifying techniques may involve a variety of hardware andsoftware tools that can create or update a map of an environment whilesimultaneously keeping track of a device's or a user's location and/ororientation within the mapped environment. SLAM may use many differenttypes of sensors to create a map and determine a device's or a user'sposition within the map.

SLAM techniques may, for example, implement optical sensors to determinea device's or a user's location, position, or orientation. Radiosincluding WiFi, Bluetooth, global positioning system (GPS), cellular orother communication devices may also be used to determine a user'slocation relative to a radio transceiver or group of transceivers (e.g.,a WiFi router or group of GPS satellites). Acoustic sensors such asmicrophone arrays or 2D or 3D sonar sensors may also be used todetermine a user's location within an environment. AR and VR devices(such as systems 1100, 1200, and 1300) may incorporate any or all ofthese types of sensors to perform SLAM operations such as creating andcontinually updating maps of a device's or a user's current environment.In at least some of the embodiments described herein, SLAM datagenerated by these sensors may be referred to as “environmental data”and may indicate a device's or a user's current environment. This datamay be stored in a local or remote data store (e.g., a cloud data store)and may be provided to a user's AR/VR device on demand.

When the user is wearing an AR headset or VR headset in a givenenvironment, the user may be interacting with other users or otherelectronic devices that serve as audio sources. In some cases, it may bedesirable to determine where the audio sources are located relative tothe user and then present the audio sources to the user as if they werecoming from the location of the audio source. The process of determiningwhere the audio sources are located relative to the user may be referredto herein as “localization,” and the process of rendering playback ofthe audio source signal to appear as if it is coming from a specificdirection may be referred to herein as “spatialization.”

Localizing an audio source may be performed in a variety of differentways. In some cases, an AR or VR headset may initiate a DOA analysis todetermine the location of a sound source. The DOA analysis may includeanalyzing the intensity, spectra, and/or arrival time of each sound atthe AR/VR device to determine the direction from which the soundoriginated. In some cases, the DOA analysis may include any suitablealgorithm for analyzing the surrounding acoustic environment in whichthe artificial reality device is located.

For example, the DOA analysis may be designed to receive input signalsfrom a microphone and apply digital signal processing algorithms to theinput signals to estimate the direction of arrival. These algorithms mayinclude, for example, delay and sum algorithms where the input signal issampled, and the resulting weighted and delayed versions of the sampledsignal are averaged together to determine a direction of arrival. Aleast mean squared (LMS) algorithm may also be implemented to create anadaptive filter. This adaptive filter may then be used to identifydifferences in signal intensity, for example, or differences in time ofarrival. These differences may then be used to estimate the direction ofarrival. In another embodiment, the DOA may be determined by convertingthe input signals into the frequency domain and selecting specific binswithin the time-frequency (TF) domain to process. Each selected TF binmay be processed to determine whether that bin includes a portion of theaudio spectrum with a direct-path audio signal. Those bins having aportion of the direct-path signal may then be analyzed to identify theangle at which a microphone array received the direct-path audio signal.The determined angle may then be used to identify the direction ofarrival for the received input signal. Other algorithms not listed abovemay also be used alone or in combination with the above algorithms todetermine DOA.

In some embodiments, different users may perceive the source of a soundas coming from slightly different locations. This may be the result ofeach user having a unique head-related transfer function (HRTF), whichmay be dictated by a user's anatomy including ear canal length and thepositioning of the ear drum. The artificial reality device may providean alignment and orientation guide, which the user may follow tocustomize the sound signal presented to the user based on their uniqueHRTF. In some embodiments, an AR or VR device may implement one or moremicrophones to listen to sounds within the user's environment. The AR orVR device may use a variety of different array transfer functions (ATFs)(e.g., any of the DOA algorithms identified above) to estimate thedirection of arrival for the sounds. Once the direction of arrival hasbeen determined, the artificial reality device may play back sounds tothe user according to the user's unique HRTF. Accordingly, the DOAestimation generated using an ATF may be used to determine the directionfrom which the sounds are to be played from. The playback sounds may befurther refined based on how that specific user hears sounds accordingto the HRTF.

In addition to or as an alternative to performing a DOA estimation, anartificial reality device may perform localization based on informationreceived from other types of sensors. These sensors may include cameras,infrared radiation (IR) sensors, heat sensors, motion sensors, globalpositioning system (GPS) receivers, or in some cases, sensor that detecta user's eye movements. For example, an artificial reality device mayinclude an eye tracker or gaze detector that determines where a user islooking. Often, a user's eyes will look at the source of a sound, ifonly briefly. Such clues provided by the user's eyes may further aid indetermining the location of a sound source. Other sensors such ascameras, heat sensors, and IR sensors may also indicate the location ofa user, the location of an electronic device, or the location of anothersound source. Any or all of the above methods may be used individuallyor in combination to determine the location of a sound source and mayfurther be used to update the location of a sound source over time.

Some embodiments may implement the determined DOA to generate a morecustomized output audio signal for the user. For instance, an acoustictransfer function may characterize or define how a sound is receivedfrom a given location. More specifically, an acoustic transfer functionmay define the relationship between parameters of a sound at its sourcelocation and the parameters by which the sound signal is detected (e.g.,detected by a microphone array or detected by a user's ear). Anartificial reality device may include one or more acoustic sensors thatdetect sounds within range of the device. A controller of the artificialreality device may estimate a DOA for the detected sounds (using, e.g.,any of the methods identified above) and, based on the parameters of thedetected sounds, may generate an acoustic transfer function that isspecific to the location of the device. This customized acoustictransfer function may thus be used to generate a spatialized outputaudio signal where the sound is perceived as coming from a specificlocation.

Indeed, once the location of the sound source or sources is known, theartificial reality device may re-render (i.e., spatialize) the soundsignals to sound as if coming from the direction of that sound source.The artificial reality device may apply filters or other digital signalprocessing that alter the intensity, spectra, or arrival time of thesound signal. The digital signal processing may be applied in such a waythat the sound signal is perceived as originating from the determinedlocation. The artificial reality device may amplify or subdue certainfrequencies or change the time that the signal arrives at each ear. Insome cases, the artificial reality device may create an acoustictransfer function that is specific to the location of the device and thedetected direction of arrival of the sound signal. In some embodiments,the artificial reality device may re-render the source signal in astereo device or multi-speaker device (e.g., a surround sound device).In such cases, separate and distinct audio signals may be sent to eachspeaker. Each of these audio signals may be altered according to auser's HRTF and according to measurements of the user's location and thelocation of the sound source to sound as if they are coming from thedetermined location of the sound source. Accordingly, in this manner,the artificial reality device (or speakers associated with the device)may re-render an audio signal to sound as if originating from a specificlocation.

Although some of various drawings illustrate a number of logical stagesin a particular order, stages which are not order dependent may bereordered and other stages may be combined or broken out. While somereordering or other groupings are specifically mentioned, others will beobvious to those of ordinary skill in the art, so the ordering andgroupings presented herein are not an exhaustive list of alternatives.Moreover, it should be recognized that the stages could be implementedin hardware, firmware, software or any combination thereof.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the scope of the claims to the precise forms disclosed. Manymodifications and variations are possible in view of the aboveteachings. The embodiments were chosen in order to best explain theprinciples underlying the claims and their practical applications, tothereby enable others skilled in the art to best use the embodimentswith various modifications as are suited to the particular usescontemplated.

It is noted that the embodiments disclosed herein can also be combinedwith any of the embodiments described in U.S. Provisional ApplicationNo. 62/647,559, filed Mar. 23, 2018, entitled “Methods, Devices, andSystems for Determining Contact On a User of a Virtual Reality and/orAugmented Reality Device;” U.S. Provisional Application No. 62/636,699,filed Feb. 28, 2018, entitled “Methods, Devices, and Systems forCreating Haptic Stimulations and Tracking Motion of a User;” and U.S.Provisional Application No. 62/614,790, filed Jan. 8, 2018, entitled“Methods, Devices, and Systems for Creating Localized Haptic Sensationson a User.”

It also is noted that the embodiments disclosed herein can also becombined with any of the embodiments described in U.S. Utility patentapplication Ser. No. 15/241,871, entitled “Methods, Devices, and Systemsfor Creating Haptic Stimulations and Tracking Motion of a User,” filedJan. 7, 2019, U.S. Utility patent application Ser. No. 16/241,890,entitled “Methods, Devices, and Systems for Determining Contact On aUser of a Virtual Reality and/or Augmented Reality Device,” filed Jan.7, 2019, and U.S. Utility patent application Ser. No. 16/241,900,entitled “Methods, Devices, and Systems for Creating Localized HapticSensations on a User,” filed Jan. 7, 2019.

What is claimed is:
 1. A method, comprising: at a wearable device that(i) includes one or more transducers and (ii) is configured to attach toan appendage of a user: receiving, by the one or more transducers, a setof signals that establish a signal pathway to the wearable device,wherein the signals in the set of signals propagate through the user'sappendage; while receiving the set of signals: determining baselinecharacteristics for the signal pathway; and sensing a change in thebaseline characteristics caused by user interaction with an affordanceof a user interface projected or perceived on the user's appendage; andin accordance with a determination that the sensed change in thebaseline characteristics for the signal pathway satisfies a contactcriterion, reporting a candidate touch event on the user's appendage toa separate electronic device, wherein the separate electronic devicecreates the user interface or is in communication with anotherelectronic device that creates the user interface.
 2. The method ofclaim 1, wherein reporting the candidate touch event comprises sendingtransducer data corresponding to the sensed change in the baselinecharacteristics to the separate electronic device.
 3. The method ofclaim 2, further comprising, at the wearable device: determining anapproximate location of the candidate touch event on the user'sappendage based, at least in part, on the sensed change in the baselinecharacteristics, wherein the transducer data sent to the separateelectronic device further comprises information indicating theapproximate location of the candidate touch event.
 4. The method ofclaim 2, wherein the transducer data sent to the separate electronicdevice also indicates an approximate location of the candidate touchevent on the user's appendage.
 5. The method of claim 2, furthercomprising: capturing, via one or more cameras of the separateelectronic device, the candidate touch event; generating, by theseparate electronic device, image data according to the capturing of thecandidate touch event; and executing, by the separate electronic device,a function associated with the affordance of the user interface afterprocessing the transducer data and the image data.
 6. The method ofclaim 1, further comprising: displaying, by the separate electronicdevice, the affordance of the user interface on the user's appendage. 7.The method of claim 1, wherein: the baseline characteristics include abaseline phase value; and sensing the change in the baselinecharacteristics for the signal pathway comprises detecting a phase valueof the signal pathway that differs from the baseline phase value.
 8. Themethod of claim 7, wherein: the contact criterion includes a phasedifference threshold; and reporting the candidate touch event isperformed in accordance with a determination that a difference betweenthe phase value and the baseline phase value satisfies the phasedifference threshold.
 9. The method of claim 1, wherein: the baselinecharacteristics include a baseline amplitude value; and sensing thechange in the baseline characteristics for the signal pathway comprisesdetecting an amplitude value of the signal pathway that differs from thebaseline amplitude value.
 10. The method of claim 9, wherein: thecontact criterion includes an amplitude difference threshold; andreporting the candidate touch event is performed in accordance with adetermination that a difference between the amplitude value and thebaseline amplitude value satisfies the amplitude difference threshold.11. The method of claim 1, wherein: the baseline characteristics includea baseline amplitude value and a baseline phase value; and sensing thechange in the baseline characteristics for the signal pathway comprisesdetecting (i) an amplitude value of the signal pathway that differs fromthe baseline amplitude value, and (ii) a phase value of the signalpathway that differs from the baseline phase value.
 12. The method ofclaim 11, wherein: the contact criterion includes an amplitudedifference threshold and a phase difference threshold; and reporting thecandidate touch event is performed in accordance with a determinationthat: (i) a difference between the amplitude value and the baselineamplitude value satisfies the amplitude difference threshold, and (ii) adifference between the phase value and the baseline phase valuesatisfies the phase difference threshold.
 13. The method of claim 1,wherein: the contact criterion includes a time threshold; sensing thechange in the baseline characteristics comprises sensing the change fora period of time; and reporting the candidate touch event is performedin accordance with a determination that the period of time satisfies thetime threshold.
 14. The method of claim 1, further comprising, at thewearable device: before receiving the set of signals: receiving aplurality predetermined values for signals characteristics, wherein eachof the predetermined values for the signals characteristics correspondsto a specific location of the appendage of the user.
 15. The method ofclaim 1, wherein the candidate touch event is selected from the groupconsisting of: a tap gesture, a press-and-hold gesture, a multi-tapgesture, a swipe gesture, a drag gesture, a pinch gesture, a pullgesture, a hover, and a twist gesture.
 16. The method of claim 1,wherein: reporting the candidate touch event comprises sending, to theseparate electronic device, data associated with the sensed change inthe signal pathway; and the separate electronic device determineswhether the user intended to interact with the affordance of the userinterface displayed on the user's appendage based, at least in part, onthe data associated with the sensed change in the signal pathway. 17.The method of claim 1, wherein the separate electronic device is anartificial-reality system selected from the group consisting of: anaugmented-reality system, a virtual-reality system, and a mixed-realitysystem.
 18. A wearable device attached to an appendage of a user, thewearable device comprising: one or more transducers; one or moreprocessors; and memory storing one or more programs, which when executedby the one or more processors cause the wearable device to: receive, bythe one or more transducers, a set of signals that establish a signalpathway to the wearable device, wherein the signals in the set ofsignals propagate through the user's appendage; while receiving the setof signals: determine baseline characteristics for the signal pathway;and sense a change in the baseline characteristics caused by userinteraction with an affordance of a user interface projected orperceived on the user's appendage; and in accordance with adetermination that the sensed change in the baseline characteristics forthe signal pathway satisfies a contact criterion, report a candidatetouch event on the user's appendage to a separate electronic device,wherein the separate electronic device creates the user interface or isin communication with another electronic device that creates the userinterface.
 19. A non-transitory computer-readable storage medium storingone or more programs configured for execution by one or more processorsof a wearable device having one or more transducers and configured toattach to an appendage of a user, the one or more programs includinginstructions, which when executed by the one or more processors, causethe wearable device to: receive, by the one or more transducers, a setof signals that establish a signal pathway to the wearable device,wherein the signals in the set of signals propagate through the user'sappendage; while receiving the set of signals: determine baselinecharacteristics for the signal pathway; and sense a change in thebaseline characteristics caused by user interaction with an affordanceof a user interface projected or perceived on the user's appendage; andin accordance with a determination that the sensed change in thebaseline characteristics for the signal pathway satisfies a contactcriterion, report a candidate touch event on the user's appendage to aseparate electronic device, wherein the separate electronic devicecreates the user interface or is in communication with anotherelectronic device that creates the user interface.